Thread: Re: Problem with dblink regression test
"Jim C. Nasby" <decibel@decibel.org> writes:
> It appears that the dblink regression test defaults to port 5432. I've
> been trying to get platypus to compile clean on HEAD and 8_0 and it's
> been failing on dblink.
There are several buildfarm machines failing like this. I think a
possible solution is for the postmaster to do putenv("PGPORT=nnn")
so that libpq instances running in postmaster children will default
to the local installation's actual port rather than some compiled-in
default port.
This is certainly not without its downsides, but if you are running
a postmaster at a nondefault port then I think you ought to be aware
that leaving dblink to choose a default port is a fragile idea.
Thoughts?
regards, tom lane
Tom Lane wrote:
> "Jim C. Nasby" <decibel@decibel.org> writes:
>
>>It appears that the dblink regression test defaults to port 5432. I've
>>been trying to get platypus to compile clean on HEAD and 8_0 and it's
>>been failing on dblink.
>
>
> There are several buildfarm machines failing like this. I think a
> possible solution is for the postmaster to do putenv("PGPORT=nnn")
> so that libpq instances running in postmaster children will default
> to the local installation's actual port rather than some compiled-in
> default port.
>
> This is certainly not without its downsides, but if you are running
> a postmaster at a nondefault port then I think you ought to be aware
> that leaving dblink to choose a default port is a fragile idea.
>
> Thoughts?
(Sorry for the slow response, I'm away from home again, this time in
South Korea)
I think most people would expect that if they don't specify a port, they
would be talking to the same postmaster that they are running under on
whatever port it is using, not some compiled in default. So your
proposal makes perfect sense to me. Then the dblink regression test
would not specify a port at all, correct?
Joe
Tom Lane said:
> "Jim C. Nasby" <decibel@decibel.org> writes:
>> It appears that the dblink regression test defaults to port 5432. I've
>> been trying to get platypus to compile clean on HEAD and 8_0 and it's
>> been failing on dblink.
>
> There are several buildfarm machines failing like this. I think a
> possible solution is for the postmaster to do putenv("PGPORT=nnn") so
> that libpq instances running in postmaster children will default to the
> local installation's actual port rather than some compiled-in default
> port.
>
> This is certainly not without its downsides, but if you are running a
> postmaster at a nondefault port then I think you ought to be aware that
> leaving dblink to choose a default port is a fragile idea.
>
This seems to be my day for getting confused.
If this diagnosis were correct, wouldn't every buildfarm member be failing
at the ContribCheck stage (if they get that far)? They all run on non
standard ports and all run the contrib installcheck suite if they can (this
is required, not optional). So if they show OK then they do not exhibit the
problem.
Also, while the PGPORT=nnnn trick looks sort of OK, we need to check it will
work on Windows - I am far from sure it will.
cheers
andrew
"Andrew Dunstan" <andrew@dunslane.net> writes:
> If this diagnosis were correct, wouldn't every buildfarm member be failing
> at the ContribCheck stage (if they get that far)?
I am way too tired right now to run down the details, but there is a
series of possibilities for the port libpq will try to connect to,
and I think what is happening is that one of the lower-priority choices
is causing it to connect to an existing installed postmaster instead of
the intended test installation. Anyone want to do the legwork to
explain this clearly?
> Also, while the PGPORT=nnnn trick looks sort of OK, we need to check it will
> work on Windows - I am far from sure it will.
Yeah, the Windows case has its own all-new set of gotchas ...
regards, tom lane
"Andrew Dunstan" <andrew@dunslane.net> writes:
> Tom Lane said:
>> There are several buildfarm machines failing like this. I think a
>> possible solution is for the postmaster to do putenv("PGPORT=nnn") so
>> that libpq instances running in postmaster children will default to the
>> local installation's actual port rather than some compiled-in default
>> port.
> If this diagnosis were correct, wouldn't every buildfarm member be failing
> at the ContribCheck stage (if they get that far)? They all run on non
> standard ports and all run the contrib installcheck suite if they can (this
> is required, not optional). So if they show OK then they do not exhibit the
> problem.
Now that I'm a little more awake ...
I think the difference between the working and not-working machines
probably has to do with dynamic-linker configuration. You have the
buildfarm builds using "configure --prefix=something
--with-pgport=something". So, the copy of libpq.so installed into
the prefix tree has the "right" default port. But on a machine with
a regular installation of Postgres, there is also going to be a copy
of libpq.so in /usr/lib or some such place ... and that copy thinks
the default port is where the regular postmaster lives (eg 5432).
When dblink.so is loaded into the backend, if the dynamic linker chooses
to resolve its requirement for libpq.so by loading /usr/lib/libpq.so,
then the wrong things happen.
In the "make check" case this is masked because pg_regress.sh has set
PGPORT in the postmaster's environment, and that will override the
compiled-in default. But of course the contrib tests only work in
"installcheck" mode.
To believe this, you have to assume that "psql" links to the correct
version (the test version) of libpq.so but dblink.so fails to do so.
So it's only an issue on platforms where "rpath" works for executables
but not for shared libraries. I haven't run down exactly which
buildfarm machines have shown this symptom --- do you know offhand?
(Thinks some more...) Another possibility is that on the failing
machines, there is a system-wide PGPORT environment variable; however,
unless you specify "-p" on the postmaster command line when you start
the "installed" postmaster, I'd expect that to change where the
postmaster puts its socket, so that's probably not the right answer.
If this is the correct explanation, then fooling with PGPORT would
mask this particular symptom, but it wouldn't fix the fundamental
problem that we're loading the wrong version of libpq.so. Eventually
that would come back to bite us (whenever dblink.so requires some
feature that doesn't exist in older libpq.so versions).
regards, tom lane
On Wed, Jun 22, 2005 at 11:45:09AM -0400, Tom Lane wrote:
> "Andrew Dunstan" <andrew@dunslane.net> writes:
> > Tom Lane said:
> >> There are several buildfarm machines failing like this. I think a
> >> possible solution is for the postmaster to do putenv("PGPORT=nnn") so
> >> that libpq instances running in postmaster children will default to the
> >> local installation's actual port rather than some compiled-in default
> >> port.
>
> > If this diagnosis were correct, wouldn't every buildfarm member be failing
> > at the ContribCheck stage (if they get that far)? They all run on non
> > standard ports and all run the contrib installcheck suite if they can (this
> > is required, not optional). So if they show OK then they do not exhibit the
> > problem.
>
> Now that I'm a little more awake ...
>
> I think the difference between the working and not-working machines
> probably has to do with dynamic-linker configuration. You have the
> buildfarm builds using "configure --prefix=something
> --with-pgport=something". So, the copy of libpq.so installed into
> the prefix tree has the "right" default port. But on a machine with
> a regular installation of Postgres, there is also going to be a copy
> of libpq.so in /usr/lib or some such place ... and that copy thinks
> the default port is where the regular postmaster lives (eg 5432).
> When dblink.so is loaded into the backend, if the dynamic linker chooses
> to resolve its requirement for libpq.so by loading /usr/lib/libpq.so,
> then the wrong things happen.
>
> In the "make check" case this is masked because pg_regress.sh has set
> PGPORT in the postmaster's environment, and that will override the
> compiled-in default. But of course the contrib tests only work in
> "installcheck" mode.
>
> To believe this, you have to assume that "psql" links to the correct
> version (the test version) of libpq.so but dblink.so fails to do so.
> So it's only an issue on platforms where "rpath" works for executables
> but not for shared libraries. I haven't run down exactly which
> buildfarm machines have shown this symptom --- do you know offhand?
>
> (Thinks some more...) Another possibility is that on the failing
> machines, there is a system-wide PGPORT environment variable; however,
> unless you specify "-p" on the postmaster command line when you start
> the "installed" postmaster, I'd expect that to change where the
> postmaster puts its socket, so that's probably not the right answer.
>
> If this is the correct explanation, then fooling with PGPORT would
> mask this particular symptom, but it wouldn't fix the fundamental
> problem that we're loading the wrong version of libpq.so. Eventually
> that would come back to bite us (whenever dblink.so requires some
> feature that doesn't exist in older libpq.so versions).
Here's the info I have for my two machines (platypus and cuckoo), both
of which are exhibiting this behavior.
I manually ran the dblink regression on platypus to see what was going
on. If I added port=5682 to the connection string, it would properly
connect to the test database. Without that it complained that the
contrib_regression database didn't exist. After adding
contrib_regression to the default postgresql cluster on that machine it
then errored out saying that there was no buildfarm user, which is true
on the default install on that machine. $PGPORT isn't set globally or in
the buildfarm user account.
ISTM there's a couple ways a buildfarm machine could pass besides what
Tom's mentioned. If the machine doesn't have a default install at all
it's possible that dblink will act differently. It's also possible that
the default install has both the contrib_regression database and the
user that's running the buildfarm.
Is there a way to confirm which libpq.so psql and/or dblink.so has
linked to? Are there any other tests I could run to shed some light on
this?
--
Jim C. Nasby, Database Consultant decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
On Tue, Jun 21, 2005 at 08:49:12PM -0700, Joe Conway wrote: > I think most people would expect that if they don't specify a port, they > would be talking to the same postmaster that they are running under on > whatever port it is using, not some compiled in default. So your > proposal makes perfect sense to me. Then the dblink regression test > would not specify a port at all, correct? Actually, the regression test currently doesn't specify a port. If it did we wouldn't have found this problem. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
"Jim C. Nasby" <decibel@decibel.org> writes:
> Is there a way to confirm which libpq.so psql and/or dblink.so has
> linked to? Are there any other tests I could run to shed some light on
> this?
On Linux you use "ldd" to find out what the linker will do with
dependencies of an executable or shared library. I don't recall the
equivalent incantation on FreeBSD or OS X but I'm sure there is one.
Note that this is very likely to depend on environment (eg
LD_LIBRARY_PATH) so make sure you do it in the same environment the
buildfarm test has.
regards, tom lane
On Jun 22, 2005, at 12:52, Tom Lane wrote: > "Jim C. Nasby" <decibel@decibel.org> writes: > >> Is there a way to confirm which libpq.so psql and/or dblink.so has >> linked to? Are there any other tests I could run to shed some >> light on >> this? > > On Linux you use "ldd" to find out what the linker will do with > dependencies of an executable or shared library. I don't recall the > equivalent incantation on FreeBSD or OS X but I'm sure there is one. On OS X, use "otool -L": $ otool -L /Library/PostgreSQL/bin/psql /Library/PostgreSQL/bin/psql: /Library/PostgreSQL/lib/libpq.3.dylib (compatibility version 3.0.0, current version 3.2.0) /usr/lib/libssl.0.9.7.dylib (compatibility version 0.9.7, current version 0.9.7) /usr/lib/libcrypto.0.9.7.dylib (compatibility version 0.9.7, current version 0.9.7) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libncurses.5.dylib (compatibility version 5.0.0, current version 5.0.0) /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 324.9.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.1) - Chris
On Mon, Jun 27, 2005 at 02:10:47PM -0500, Jim C. Nasby wrote: > http://stats.distributed.net/~buildfarm/libcheck.log. Note that Tom's > theory is correct: psql is linking against the buildfarm libpq while > dblink is linking against the system one. BTW, after looking through that logfile, it appears that all the libraries are linking against the system libraries instead of the buildfarm/*/inst libraries, so this isn't specifically a dblink or even contrib issue. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
Jim C. Nasby wrote: >On Mon, Jun 27, 2005 at 02:10:47PM -0500, Jim C. Nasby wrote: > > >>http://stats.distributed.net/~buildfarm/libcheck.log. Note that Tom's >>theory is correct: psql is linking against the buildfarm libpq while >>dblink is linking against the system one. >> >> > >BTW, after looking through that logfile, it appears that all the >libraries are linking against the system libraries instead of the >buildfarm/*/inst libraries, so this isn't specifically a dblink or even >contrib issue. > > I only saw problems with libecpg.so, libecpg_compat.so and dblink.so cheers andrew
On Mon, Jun 27, 2005 at 03:44:41PM -0400, Andrew Dunstan wrote: > > > Jim C. Nasby wrote: > > >On Mon, Jun 27, 2005 at 02:10:47PM -0500, Jim C. Nasby wrote: > > > > > >>http://stats.distributed.net/~buildfarm/libcheck.log. Note that Tom's > >>theory is correct: psql is linking against the buildfarm libpq while > >>dblink is linking against the system one. > >> > >> > > > >BTW, after looking through that logfile, it appears that all the > >libraries are linking against the system libraries instead of the > >buildfarm/*/inst libraries, so this isn't specifically a dblink or even > >contrib issue. > > > > > > I only saw problems with libecpg.so, libecpg_compat.so and dblink.so AFAICT those are the only libraries that link to libpq or any other postgresql library, so yes, they'd be the only ones with a problem. :) BTW, anyone have any idea why my other two emails haven't gone out to the list yet? Is it because of their size? -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
I have no clue why the mailling list is eating my original messages, unless it's because I attached a diff to them... in any case, applying http://stats.distributed.net/~buildfarm/patch provides a listing of what all the binaries and libraries in a buildfarm install are linking against. I couldn't figure out a decent way to send that info to pgbuildfarm.org, but http://stats.distributed.net/~buildfarm/libcheck.log is that info for a run. Based on the logfile, it looks like Tom's guess is correct that psql (and other binaries) are linking to the buildfarm libraries, while dblink.so (and other libraries) are linking against the system postgresql libraries. If someone wants to point me in the right direction I'll try and fix this, since I'm guessing it's just a make issue (or maybe a buildfarm issue). Actually, looking at my config (http://stats.distributed.net/~buildfarm/build-farm.conf), could the --with-libraries=/usr/local/lib be the issue, and if so, what's the proper way to handle system libraries (like libintl) living in /usr/local/lib and not /usr/lib? -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
Jim, you should have a file <buildroot>/HEAD/lastrun-logs/make.log that shows the link steps for the libraries. Can you either put that file somewhere we can look at it or extract the relevant lines and post in a reply? thanks andrew Jim C. Nasby wrote: >I have no clue why the mailling list is eating my original messages, >unless it's because I attached a diff to them... in any case, applying >http://stats.distributed.net/~buildfarm/patch provides a listing of what >all the binaries and libraries in a buildfarm install are linking >against. I couldn't figure out a decent way to send that info to >pgbuildfarm.org, but >http://stats.distributed.net/~buildfarm/libcheck.log is that info for a >run. Based on the logfile, it looks like Tom's guess is correct that >psql (and other binaries) are linking to the buildfarm libraries, while >dblink.so (and other libraries) are linking against the system >postgresql libraries. > >If someone wants to point me in the right direction I'll try and fix >this, since I'm guessing it's just a make issue (or maybe a buildfarm >issue). > >Actually, looking at my config >(http://stats.distributed.net/~buildfarm/build-farm.conf), could the >--with-libraries=/usr/local/lib be the issue, and if so, what's the >proper way to handle system libraries (like libintl) living in >/usr/local/lib and not /usr/lib? > >
All the logs for the most recent run against HEAD are now at http://stats.distributed.net/~buildfarm/ On Tue, Jun 28, 2005 at 08:30:44AM -0400, Andrew Dunstan wrote: > > Jim, > > you should have a file <buildroot>/HEAD/lastrun-logs/make.log that shows > the link steps for the libraries. Can you either put that file somewhere > we can look at it or extract the relevant lines and post in a reply? > > thanks > > andrew > > Jim C. Nasby wrote: > > >I have no clue why the mailling list is eating my original messages, > >unless it's because I attached a diff to them... in any case, applying > >http://stats.distributed.net/~buildfarm/patch provides a listing of what > >all the binaries and libraries in a buildfarm install are linking > >against. I couldn't figure out a decent way to send that info to > >pgbuildfarm.org, but > >http://stats.distributed.net/~buildfarm/libcheck.log is that info for a > >run. Based on the logfile, it looks like Tom's guess is correct that > >psql (and other binaries) are linking to the buildfarm libraries, while > >dblink.so (and other libraries) are linking against the system > >postgresql libraries. > > > >If someone wants to point me in the right direction I'll try and fix > >this, since I'm guessing it's just a make issue (or maybe a buildfarm > >issue). > > > >Actually, looking at my config > >(http://stats.distributed.net/~buildfarm/build-farm.conf), could the > >--with-libraries=/usr/local/lib be the issue, and if so, what's the > >proper way to handle system libraries (like libintl) living in > >/usr/local/lib and not /usr/lib? > > > > -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
Jim C. Nasby wrote: >All the logs for the most recent run against HEAD are now at >http://stats.distributed.net/~buildfarm/ > > > > A quick look shows that when you use --with-libraries=/foo/bar the generated link line for libraries says -L/foo/bar -lpq and it should probably be the other way around (as it is for the executables). So I suspect we need some makefile tuning. cheers andrew
On Tue, Jun 28, 2005 at 02:28:11PM -0400, Andrew Dunstan wrote: > > > Jim C. Nasby wrote: > > >All the logs for the most recent run against HEAD are now at > >http://stats.distributed.net/~buildfarm/ > > > > > > > > > > A quick look shows that when you use --with-libraries=/foo/bar the > generated link line for libraries says > > -L/foo/bar -lpq > > and it should probably be the other way around (as it is for the > executables). > > So I suspect we need some makefile tuning. You were correct. This patch fixes it: Index: Makefile.shlib =================================================================== RCS file: /projects/cvsroot/pgsql/src/Makefile.shlib,v retrieving revision 1.90 diff -c -r1.90 Makefile.shlib *** Makefile.shlib 20 Nov 2004 21:13:04 -0000 1.90 --- Makefile.shlib 29 Jun 2005 00:21:10 -0000 *************** *** 240,246 **** SHLIB_LINK += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86 endif ! SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK) ifeq ($(enable_rpath), yes) SHLIB_LINK += $(rpath) endif --- 240,246 ---- SHLIB_LINK += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86 endif ! SHLIB_LINK := $(SHLIB_LINK) $(filter -L%, $(LDFLAGS)) ifeq ($(enable_rpath), yes) SHLIB_LINK += $(rpath) endif -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"
Patch applied. Thanks. --------------------------------------------------------------------------- Jim C. Nasby wrote: > On Tue, Jun 28, 2005 at 02:28:11PM -0400, Andrew Dunstan wrote: > > > > > > Jim C. Nasby wrote: > > > > >All the logs for the most recent run against HEAD are now at > > >http://stats.distributed.net/~buildfarm/ > > > > > > > > > > > > > > > > A quick look shows that when you use --with-libraries=/foo/bar the > > generated link line for libraries says > > > > -L/foo/bar -lpq > > > > and it should probably be the other way around (as it is for the > > executables). > > > > So I suspect we need some makefile tuning. > > You were correct. This patch fixes it: > Index: Makefile.shlib > =================================================================== > RCS file: /projects/cvsroot/pgsql/src/Makefile.shlib,v > retrieving revision 1.90 > diff -c -r1.90 Makefile.shlib > *** Makefile.shlib 20 Nov 2004 21:13:04 -0000 1.90 > --- Makefile.shlib 29 Jun 2005 00:21:10 -0000 > *************** > *** 240,246 **** > SHLIB_LINK += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86 > endif > > ! SHLIB_LINK := $(filter -L%, $(LDFLAGS)) $(SHLIB_LINK) > ifeq ($(enable_rpath), yes) > SHLIB_LINK += $(rpath) > endif > --- 240,246 ---- > SHLIB_LINK += -ltermcap -lstdc++.r4 -lbind -lsocket -L/boot/develop/lib/x86 > endif > > ! SHLIB_LINK := $(SHLIB_LINK) $(filter -L%, $(LDFLAGS)) > ifeq ($(enable_rpath), yes) > SHLIB_LINK += $(rpath) > endif > > -- > Jim C. Nasby, Database Consultant decibel@decibel.org > Give your computer some brain candy! www.distributed.net Team #1828 > > Windows: "Where do you want to go today?" > Linux: "Where do you want to go tomorrow?" > FreeBSD: "Are you guys coming, or what?" > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073