Thread: Runtime SHAREDIR for testing CREATE EXTENSION
I'm trying to understand what options I have to test "CREATE EXTENSION" w/out installing the extension files in their final destination. Could not find a way to set SHAREDIR from within psql, nor a way to specify it in initdb call. Am I missing it something ? PS: please include my address in replies as I have mail delivery disabled from the list. Thank you. --strk; ,------o-. | __/ | Delivering high quality PostGIS 2.0 ! | / 2.0 | http://strk.keybit.net `-o------'
Sandro Santilli <strk@keybit.net> writes:
> I'm trying to understand what options I have to test "CREATE EXTENSION"
> w/out installing the extension files in their final destination.
There aren't any. Generally speaking, if you want to be testing an
extension, you should be doing it in a test installation anyway;
so I don't see what's the objection to sticking the files into the
installation's real SHAREDIR.
regards, tom lane
On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote: > Sandro Santilli <strk@keybit.net> writes: > > I'm trying to understand what options I have to test "CREATE EXTENSION" > > w/out installing the extension files in their final destination. > > There aren't any. Generally speaking, if you want to be testing an > extension, you should be doing it in a test installation anyway; > so I don't see what's the objection to sticking the files into the > installation's real SHAREDIR. We do a staged installation of the extension library and scripts in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL. It works fine for testing the extension directly sourcing the scripts. It's only "CREATE EXTENSION" that doesn't play nicely with that setup. --strk;
Sandro Santilli <strk@keybit.net> writes: > On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote: >> Sandro Santilli <strk@keybit.net> writes: >> > I'm trying to understand what options I have to test "CREATE EXTENSION" >> > w/out installing the extension files in their final destination. >> >> There aren't any. Generally speaking, if you want to be testing an >> extension, you should be doing it in a test installation anyway; >> so I don't see what's the objection to sticking the files into the >> installation's real SHAREDIR. > > We do a staged installation of the extension library and scripts > in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL. > > It works fine for testing the extension directly sourcing the scripts. > It's only "CREATE EXTENSION" that doesn't play nicely with that setup. Please see the "inline extension" thread where answers to your problem have been discussed. https://commitfest.postgresql.org/action/patch_view?id=746 Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Tue, Feb 21, 2012 at 10:34:42PM +0100, Dimitri Fontaine wrote: > Sandro Santilli <strk@keybit.net> writes: > > On Tue, Feb 21, 2012 at 10:21:17AM -0500, Tom Lane wrote: > >> Sandro Santilli <strk@keybit.net> writes: > >> > I'm trying to understand what options I have to test "CREATE EXTENSION" > >> > w/out installing the extension files in their final destination. > >> > >> There aren't any. Generally speaking, if you want to be testing an > >> extension, you should be doing it in a test installation anyway; > >> so I don't see what's the objection to sticking the files into the > >> installation's real SHAREDIR. > > > > We do a staged installation of the extension library and scripts > > in an ad-hoc dir which does not match the built-in SHAREDIR of PostgreSQL. > > > > It works fine for testing the extension directly sourcing the scripts. > > It's only "CREATE EXTENSION" that doesn't play nicely with that setup. > > Please see the "inline extension" thread where answers to your problem > have been discussed. > > https://commitfest.postgresql.org/action/patch_view?id=746 I'm not really looking for "inline extensions". I do want to install the extension objects somewhere, just NOT in the PostgreSQL builtin SHAREDIR but in an arbitrary staging directory to use for QA the extension before distribution. --strk; ,------o-. | __/ | Delivering high quality PostGIS 2.0 ! | / 2.0 | http://strk.keybit.net `-o------'
On Tue, Feb 21, 2012 at 1:34 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: > Sandro Santilli <strk@keybit.net> writes: > Please see the "inline extension" thread where answers to your problem > have been discussed. I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of no help here. Can you tell us why alternative approaches that don't require adding a knob (such as copying/symlinking of compiled artifacts) is such a huge pain for you? -- fdr
Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash> > I'm not really looking for "inline extensions". > I do want to install the extension objects somewhere, just NOT > in the PostgreSQL builtin SHAREDIR but in an arbitrary staging > directory to use for QA the extension before distribution. We have the same problem with testing extensions at build-time in the Debian packages. The server's SHAREDIR /usr/share/postgresql/... is only writable by root, while the build is running as buildd user, so there is no way to do "create extension whatimbuildingrightnow" to be able to run regression tests, even if this is a cluster I have just created with initdb. Christoph -- cb@df7cb.de | http://www.df7cb.de/
Daniel Farina <daniel@heroku.com> writes: > I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of > no help here. Sorry about the sloppy mention, one of the things under discussion (but out of scope for 9.2) in the inline extension thread is about where to store the shared objects and extension files so that we have a real per-database objects here, rather than per major version deployments that happen to be able to impact several clusters and databases in different and subtle ways. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Fri, Feb 24, 2012 at 09:59:12AM +0100, Christoph Berg wrote: > Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash> > > I'm not really looking for "inline extensions". > > I do want to install the extension objects somewhere, just NOT > > in the PostgreSQL builtin SHAREDIR but in an arbitrary staging > > directory to use for QA the extension before distribution. > > We have the same problem with testing extensions at build-time in the > Debian packages. The server's SHAREDIR /usr/share/postgresql/... is > only writable by root, while the build is running as buildd user, so > there is no way to do "create extension whatimbuildingrightnow" to be > able to run regression tests, even if this is a cluster I have just > created with initdb. Exactly. We don't initdb with PostGIS regression testing framework but I've considered doing it for this specific case and it stroke me that even then we couldn't control SHAREDIR. --strk; ,------o-. | __/ | Delivering high quality PostGIS 2.0 ! | / 2.0 | http://strk.keybit.net - http://vizzuality.com`-o------'
On Thu, Feb 23, 2012 at 06:53:05PM -0800, Daniel Farina wrote: > On Tue, Feb 21, 2012 at 1:34 PM, Dimitri Fontaine > <dimitri@2ndquadrant.fr> wrote: > > Sandro Santilli <strk@keybit.net> writes: > > Please see the "inline extension" thread where answers to your problem > > have been discussed. > > I'm pretty sure Sandro is hacking PostGIS, so inline extensions are of > no help here. > > Can you tell us why alternative approaches that don't require adding a > knob (such as copying/symlinking of compiled artifacts) is such a huge > pain for you? It's not specifically a pain for me, but it breaks the "tests can be run in a staged environment" idiom. See Christoph Berg mail, that's exactly the same issue. --strk; ,------o-. | __/ | Delivering high quality PostGIS 2.0 ! | / 2.0 | http://strk.keybit.net - http://vizzuality.com`-o------'
Christoph Berg <cb@df7cb.de> writes:
> Re: Sandro Santilli 2012-02-22 <20120222101656.GB6125@gnash>
>> I'm not really looking for "inline extensions".
>> I do want to install the extension objects somewhere, just NOT
>> in the PostgreSQL builtin SHAREDIR but in an arbitrary staging
>> directory to use for QA the extension before distribution.
> We have the same problem with testing extensions at build-time in the
> Debian packages. The server's SHAREDIR /usr/share/postgresql/... is
> only writable by root, while the build is running as buildd user, so
> there is no way to do "create extension whatimbuildingrightnow" to be
> able to run regression tests, even if this is a cluster I have just
> created with initdb.
This seems like nonsense. If the build process has installed the
software, you surely have got permissions to write in that directory.
If you haven't installed the software, you need to do testing in a
temporary installation per "make check", and we are able to test
extensions that way too.
regards, tom lane
On fre, 2012-02-24 at 11:53 -0500, Tom Lane wrote: > > We have the same problem with testing extensions at build-time in > the > > Debian packages. The server's SHAREDIR /usr/share/postgresql/... is > > only writable by root, while the build is running as buildd user, so > > there is no way to do "create extension whatimbuildingrightnow" to > be > > able to run regression tests, even if this is a cluster I have just > > created with initdb. > > This seems like nonsense. If the build process has installed the > software, you surely have got permissions to write in that directory. The build process just installs the software in a fake root where it will be wrapped up by the packaging software. > If you haven't installed the software, you need to do testing in a > temporary installation per "make check", and we are able to test > extensions that way too. > That looks like the right answer.
On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote: > We don't initdb with PostGIS regression testing framework > but I've considered doing it for this specific case and it stroke me > that even then we couldn't control SHAREDIR. I would always create a new instance using initdb for test runs. It's easy and avoids all these problems.
On Fri, Feb 24, 2012 at 10:21 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote: >> We don't initdb with PostGIS regression testing framework >> but I've considered doing it for this specific case and it stroke me >> that even then we couldn't control SHAREDIR. > > I would always create a new instance using initdb for test runs. It's > easy and avoids all these problems. Having been in this position once before in a different but similar situation, there's one big caveat: initdb is *really* slow, so it is really painful for people who write Postgres-linked code that is compiled separately, whereby the initdb binary is stable. The frustration is not unlike when I have to run ./configure, except I had to do it all the time, every time. A ccache-like solution has worked well for at least one person I know, and wasn't hard to implement(?) for personal use. But by hook or crook, a fresh database is the "right" thing for sure. -- fdr
On 02/24/2012 06:31 PM, Daniel Farina wrote: > On Fri, Feb 24, 2012 at 10:21 AM, Peter Eisentraut<peter_e@gmx.net> wrote: >> On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote: >>> We don't initdb with PostGIS regression testing framework >>> but I've considered doing it for this specific case and it stroke me >>> that even then we couldn't control SHAREDIR. >> I would always create a new instance using initdb for test runs. It's >> easy and avoids all these problems. > Having been in this position once before in a different but similar > situation, there's one big caveat: initdb is *really* slow, so it is > really painful for people who write Postgres-linked code that is > compiled separately, whereby the initdb binary is stable. The > frustration is not unlike when I have to run ./configure, except I had > to do it all the time, every time. Really? Here's what I just got on a severely under-resourced SL6 VM: [andrew@xena ~]$ time /usr/pgsql-9.1/bin/initdb -A ident foo > /dev/null real 0m1.441s user 0m1.131s sys 0m0.309s 1.5s doesn't seem terribly slow. cheers andrew
On Fri, Feb 24, 2012 at 6:31 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > Really? Here's what I just got on a severely under-resourced SL6 VM: > > 1.5s doesn't seem terribly slow. You are right. Come to think of it, I do seem to recall that initdb got some speed improvements; these were in 8.3 days, I think. Here are my numbers for comparison, in some various forms: env time /usr/lib/postgresql/9.1/bin/initdb /tmp/foo 0.94user 0.28system 0:01.87elapsed 64%CPU (0avgtext+0avgdata 79920maxresident)k 14288inputs+85144outputs (36major+66622minor)pagefaults 0swaps env time eatmydata /usr/lib/postgresql/9.1/bin/initdb /tmp/foo 0.90user 0.27system 0:01.26elapsed 92%CPU (0avgtext+0avgdata 79984maxresident)k 856inputs+85144outputs (0major+67202minor)pagefaults 0swaps env time /usr/lib/postgresql/9.1/bin/initdb /dev/shm/blah 0.90user 0.19system 0:01.15elapsed 95%CPU (0avgtext+0avgdata 79984maxresident)k 0inputs+0outputs (0major+67192minor)pagefaults 0swaps Retracted. -- fdr
On Fri, Feb 24, 2012 at 3:31 PM, Daniel Farina <daniel@heroku.com> wrote: > Having been in this position once before in a different but similar > situation, there's one big caveat: initdb is *really* slow, so it is > really painful for people who write Postgres-linked code that is > compiled separately, whereby the initdb binary is stable. The > frustration is not unlike when I have to run ./configure, except I had > to do it all the time, every time. > > A ccache-like solution has worked well for at least one person I know, > and wasn't hard to implement(?) for personal use. Being the person in question: I don't have the full context so I'm not sure how useful this will be, but here's hoping it's not just line noise. I'm probably not at liberty to share the original code, but it was a fairly trivial shell script that did more or less the following: 1. Capture all arguments (to pass to initdb itself) except the path, and figure out the path to the cluster (via looking for -D/--pgdata/$PGDATA) 2. Create a hash of the concatenation of the arguments (again, sans cluster path) 3. Create a "template" path based on a local cache directory and the argument hash 4. a) If the directory with this path did not exist, run initdb with -D set to this and the rest of the arguments as captured; then copy to *actual* path b) If it does exist, that's because you've run initdb with these arguments before: just copy that to the cluster path you captured in step (1) instead of starting fresh This was not terribly robust (e.g., I remember that Dan had pointed out several shortcomings), but none of these were practical issues for me, and I never had a problem. There is some more work around capturing the cluster path in the initdb output and mangling it back from the template version into the original (so the output of the cached initdb looks like output of a real initdb), but that's not a big deal. The approach could probably be made solid enough for general-purpose use.
Re: Peter Eisentraut 2012-02-24 <1330107599.32452.15.camel@vanquo.pezone.net> > On fre, 2012-02-24 at 11:53 -0500, Tom Lane wrote: > > > We have the same problem with testing extensions at build-time in > > the > > > Debian packages. The server's SHAREDIR /usr/share/postgresql/... is > > > only writable by root, while the build is running as buildd user, so > > > there is no way to do "create extension whatimbuildingrightnow" to > > be > > > able to run regression tests, even if this is a cluster I have just > > > created with initdb. > > > > This seems like nonsense. If the build process has installed the > > software, you surely have got permissions to write in that directory. > > The build process just installs the software in a fake root where it > will be wrapped up by the packaging software. > > > If you haven't installed the software, you need to do testing in a > > temporary installation per "make check", and we are able to test > > extensions that way too. > > > That looks like the right answer. Well, I'm trying to invoke the extension's "make check" target at extension build time. I do have a temporary installation I own somehwere in my $HOME, but that is still trying to find extensions in /usr/share/postgresql/9.1/extension/*.control, because I am using the system's postgresql version. The build process is not running as root, so I cannot do an install of the extension to its final location. Still it would be nice to run regression tests. All that seems to be missing is the ability to put extension_control_path = /home/buildd/tmp/extension into the postgresql.conf of the temporary PG installation, or some other way like "CREATE EXTENSION foobar WITH CONTROL '/home/buildd/...'. Christoph -- cb@df7cb.de | http://www.df7cb.de/
On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote: > Well, I'm trying to invoke the extension's "make check" target at > extension build time. I do have a temporary installation I own > somehwere in my $HOME, but that is still trying to find extensions in > /usr/share/postgresql/9.1/extension/*.control, because I am using the > system's postgresql version. The build process is not running as root, > so I cannot do an install of the extension to its final location. > Still it would be nice to run regression tests. All that seems to be > missing is the ability to put > > extension_control_path = /home/buildd/tmp/extension > > into the postgresql.conf of the temporary PG installation, or some > other way like "CREATE EXTENSION foobar WITH CONTROL > '/home/buildd/...'. Yeah, of course, the extension path is not related to the data directory. So we do need some kind of path setting, just like dynamic_library_path.
On Sun, Feb 26, 2012 at 10:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote: >> Well, I'm trying to invoke the extension's "make check" target at >> extension build time. I do have a temporary installation I own >> somehwere in my $HOME, but that is still trying to find extensions in >> /usr/share/postgresql/9.1/extension/*.control, because I am using the >> system's postgresql version. The build process is not running as root, >> so I cannot do an install of the extension to its final location. >> Still it would be nice to run regression tests. All that seems to be >> missing is the ability to put >> >> extension_control_path = /home/buildd/tmp/extension >> >> into the postgresql.conf of the temporary PG installation, or some >> other way like "CREATE EXTENSION foobar WITH CONTROL >> '/home/buildd/...'. > > Yeah, of course, the extension path is not related to the data > directory. So we do need some kind of path setting, just like > dynamic_library_path. That logic seems sound to me, so +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Feb 24, 2012 at 08:21:05PM +0200, Peter Eisentraut wrote: > On fre, 2012-02-24 at 17:26 +0100, Sandro Santilli wrote: > > We don't initdb with PostGIS regression testing framework > > but I've considered doing it for this specific case and it stroke me > > that even then we couldn't control SHAREDIR. > > I would always create a new instance using initdb for test runs. It's > easy and avoids all these problems. Doesn't avoid the SHAREDIR problem, that's what I'm saying. SHAREDIR is a compile-time setting in PostgreSQL. --strk;
On Sun, Feb 26, 2012 at 09:50:04PM -0500, Robert Haas wrote: > On Sun, Feb 26, 2012 at 10:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > > On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote: > >> Well, I'm trying to invoke the extension's "make check" target at > >> extension build time. I do have a temporary installation I own > >> somehwere in my $HOME, but that is still trying to find extensions in > >> /usr/share/postgresql/9.1/extension/*.control, because I am using the > >> system's postgresql version. The build process is not running as root, > >> so I cannot do an install of the extension to its final location. > >> Still it would be nice to run regression tests. All that seems to be > >> missing is the ability to put > >> > >> extension_control_path = /home/buildd/tmp/extension > >> > >> into the postgresql.conf of the temporary PG installation, or some > >> other way like "CREATE EXTENSION foobar WITH CONTROL > >> '/home/buildd/...'. > > > > Yeah, of course, the extension path is not related to the data > > directory. So we do need some kind of path setting, just like > > dynamic_library_path. > > That logic seems sound to me, so +1. +1 here as well. I may attempt to produce a patch if this gets consensus. --strk; ,------o-. | __/ | Delivering high quality PostGIS 2.0 ! | / 2.0 | http://strk.keybit.net - http://vizzuality.com`-o------'
On Sun, Feb 26, 2012 at 7:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > On lör, 2012-02-25 at 14:21 +0100, Christoph Berg wrote: >> Well, I'm trying to invoke the extension's "make check" target at >> extension build time. I do have a temporary installation I own >> somehwere in my $HOME, but that is still trying to find extensions in >> /usr/share/postgresql/9.1/extension/*.control, because I am using the >> system's postgresql version. The build process is not running as root, >> so I cannot do an install of the extension to its final location. >> Still it would be nice to run regression tests. All that seems to be >> missing is the ability to put >> >> extension_control_path = /home/buildd/tmp/extension >> >> into the postgresql.conf of the temporary PG installation, or some >> other way like "CREATE EXTENSION foobar WITH CONTROL >> '/home/buildd/...'. > > Yeah, of course, the extension path is not related to the data > directory. So we do need some kind of path setting, just like > dynamic_library_path. A thought that may or may not be related: I'd really like to support libraries (C or otherwise) of multiple versions at the same time, when the underlying library permits. The critical use case is contribs that iterate on their storage format, which right now is really painful. While perhaps contribs should use a version byte or something and be backwards-compatible forever, to some extent that is clearly impossible and sometimes undesirable for contribs that emit small on-disk representations, and would rather use, say, the type system to disambiguate what operators fit with what data. I've been struggling to think of a more graceful way to load the correct version of extra dependencies (including non-PG intrinsic things like libgeos or libproj) when new database replicas are created, as well as helping people with upgrade issues. Is this tunable possibly related to a solution for this? -- fdr
On tis, 2012-02-28 at 11:00 -0800, Daniel Farina wrote: > I'd really like to support libraries (C or otherwise) of multiple > versions at the same time, when the underlying library permits. What's preventing you from doing that now? You need to name all the symbols differently, of course.
On Fri, Mar 2, 2012 at 10:37 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > On tis, 2012-02-28 at 11:00 -0800, Daniel Farina wrote: >> I'd really like to support libraries (C or otherwise) of multiple >> versions at the same time, when the underlying library permits. > > What's preventing you from doing that now? You need to name all the > symbols differently, of course. That's the problem: not really practical in a wider ecosystem of C libraries, especially if the library produces multiple versions. (Or, not practical unless someone writes some credible symbol-version-mangling-magic) But is it unsurmountable? -- dlsym returns a function pointer, and one would build up the operator table for the version of the extension at hand, so one might have ltree "version 1.01" and ltree "version 2.3" fields in the same database. -- fdr
On Fri, Mar 2, 2012 at 1:53 PM, Daniel Farina <daniel@heroku.com> wrote: > On Fri, Mar 2, 2012 at 10:37 AM, Peter Eisentraut <peter_e@gmx.net> wrote: >> On tis, 2012-02-28 at 11:00 -0800, Daniel Farina wrote: >>> I'd really like to support libraries (C or otherwise) of multiple >>> versions at the same time, when the underlying library permits. >> >> What's preventing you from doing that now? You need to name all the >> symbols differently, of course. > > That's the problem: not really practical in a wider ecosystem of C > libraries, especially if the library produces multiple versions. (Or, > not practical unless someone writes some credible > symbol-version-mangling-magic) > > But is it unsurmountable? -- dlsym returns a function pointer, and one > would build up the operator table for the version of the extension at > hand, so one might have ltree "version 1.01" and ltree "version 2.3" > fields in the same database. That might be possible, but it seems largely unrelated to the topic of this thread, unless I am missing something. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Mar 2, 2012 at 1:50 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> But is it unsurmountable? -- dlsym returns a function pointer, and one >> would build up the operator table for the version of the extension at >> hand, so one might have ltree "version 1.01" and ltree "version 2.3" >> fields in the same database. > > That might be possible, but it seems largely unrelated to the topic of > this thread, unless I am missing something. It might be, and might not be, because if we start getting into where to put shared objects for functionality linked to the database (rather just letting the dynamic linker do "whatever") it could crash head-on into data field versioning and replica issues. If someone thinks about it for a few minutes and says "nope, orthogonal," I'm without objection, but I did want to raise this spectre to other minds because it seems to be on the edge of that maelstrom. I also look at the inability for pg_upgrade to help when certain contribs are used between versions and can only say "phew, I'm glad that doesn't affect most people" (a sad flip side: people aren't using contribs and database extension as much as I like, but things are moving towards more of that and not less) because then the expected duration of the upgrade process becomes a very ugly compatibility matrix that nobody wants to see...somewhat dissatisfying for a service provider. Yet, formats can and will change requiring a lot of gradual I/O, and a version byte is not always practical, so the best option I can think of is to support multiple versions of formats, operators and types at the same time, and then gradually rewrite things. I'm not intending to open discussion about any of that right now (unless someone else is interested and wants to start a new thread), but I wanted to provide insight as to to why I'm especially preoccupied about this. -- fdr