Thread: HS/SR on AIX
I think I've been able to reproduce the issue floating around with streaming replication on AIX. LOG: could not bind IPv6 socket: The socket name is already in use. HINT: Is another postmaster already running on port 5433? If not, wait a few seconds and retry. LOG: database system was shut down in recovery at 2010-08-24 21:08:37 UTC LOG: entering standby mode cp: cannot stat `/opt/rg/data_tb3/steve/wals/000000010000000000000001': A file or directory in the path name does not exist. LOG: redo starts at 0/1000020 LOG: record with zero length at 0/1012280 cp: cannot stat `/opt/rg/data_tb3/steve/wals/000000010000000000000001': A file or directory in the path name does not exist. FATAL: could not load library "/opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so": 0509-022 Cannot load module /opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so. 0509-150 Dependent module libpq.a(libpq.so.5)could not be loaded. 0509-022 Cannot load module libpq.a(libpq.so.5). 0509-026 System error: A fileor directory in the path name does not exist. 0509-022 Cannot load module /opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so. 0509-150 Dependent module /opt/dbs/pgsql9-beta2/lib/libpqwalreceiver.so could not be loaded. This worked fine with beta2 but now seems to be an issue on beta4. If I do export LIBPATH=/opt/dbs/pgsql9-beta2/lib/ before starting the standby postmaster then it seems to work. I haven't yet tried running truss to try to look at where it is looking for libpq from. liblibpqwalreceiver is being linked with this gcc -maix64 -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -g -o libpqwalreceiver.so liblibpqwalreceiver.a -Wl,-bE:liblibpqwalreceiver.exp -L../../../../src/port -Wl,-bmaxdata:0x80000000,-bbigtoc -L/opt/freeware/lib -Wl,-blibpath:/opt/dbs/pgsql9-beta2/lib:/opt/freeware/lib:/usr/lib:/lib -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -L../../../../src/interfaces/libpq -lpq -Wl,-bI:../../../../src/backend/postgres.imp I'll try to look into this a bit more tomorrow or thursday. -- Steve Singer Afilias Canada Data Services Developer 416-673-1142
Steve Singer <ssinger@ca.afilias.info> writes: > I think I've been able to reproduce the issue floating around with > streaming replication on AIX. Excellent, because we weren't getting much from the original reporter. > This worked fine with beta2 but now seems to be an issue on beta4. > If I do > export LIBPATH=/opt/dbs/pgsql9-beta2/lib/ > before starting the standby postmaster then it seems to work. Fascinating. That seems to prove that it's an rpath problem. My first guess is that the relevant change between beta2 and beta4 is my LDFLAGS changes. See http://archives.postgresql.org/pgsql-committers/2010-07/msg00060.php and following commits. regards, tom lane
Tom Lane wrote: > Steve Singer <ssinger@ca.afilias.info> writes: >> I think I've been able to reproduce the issue floating around with >> streaming replication on AIX. > > Excellent, because we weren't getting much from the original reporter. I'm withdrawing my comment, today on a clean install of the binaries I am not able to reproduce any of this. Today with beta4 I can stream replication to the standby and bring the standby up to read-write without issues. Yesterday when I had put beta4 on top of beta2 without explicitly deleting all of the beta2 binaries/libraries first. I'm thinking maybe some portions of beta2 where still laying around. It is also possible that an old version of a shared library was still sitting in memory and was being picked up by the newer postgresql (man slibclean on AIX) I will do another clean build from the beta4 source tar to confirm that I'm not still having the issue but I'm thinking the original reporter might have done something similar and had some old artifacts laying around. > >> This worked fine with beta2 but now seems to be an issue on beta4. > >> If I do >> export LIBPATH=/opt/dbs/pgsql9-beta2/lib/ >> before starting the standby postmaster then it seems to work. > > Fascinating. That seems to prove that it's an rpath problem. My > first guess is that the relevant change between beta2 and beta4 is > my LDFLAGS changes. See > http://archives.postgresql.org/pgsql-committers/2010-07/msg00060.php > and following commits. > > regards, tom lane > -- Steve Singer Afilias Canada Data Services Developer 416-673-1142
Steve Singer wrote: > Tom Lane wrote: >> Steve Singer <ssinger@ca.afilias.info> writes: >>> I think I've been able to reproduce the issue floating around with >>> streaming replication on AIX. >> > > I will do another clean build from the beta4 source tar to confirm that > I'm not still having the issue but I'm thinking the original reporter > might have done something similar and had some old artifacts laying around. > A clean build from the beta4 source tarball where I'm careful to install into a clean (ie no old beta2 artifacts laying around waiting to be overwritten) isn't reproducing the issue. I'm happy to try other things if people suggest them (or if the original reporter is still getting this after making sure he cleans up old files first) but I'm thinking that was the issue. -- Steve Singer Afilias Canada Data Services Developer 416-673-1142
On Thu, Aug 26, 2010 at 12:45 AM, Steve Singer <ssinger@ca.afilias.info> wrote: > A clean build from the beta4 source tarball where I'm careful to install > into a clean (ie no old beta2 artifacts laying around waiting to be > overwritten) isn't reproducing the issue. > > I'm happy to try other things if people suggest them (or if the original > reporter is still getting this after making sure he cleans up old files > first) but I'm thinking that was the issue. Thanks for the report! Alanoly, could you do a clean install and try the test again? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
Fujii, All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installedin different and unique directories. Regards. Alanoly. -----Original Message----- From: Fujii Masao [mailto:masao.fujii@gmail.com] Sent: Thursday, August 26, 2010 1:44 AM To: Steve Singer Cc: Tom Lane; PostgreSQL-development Hackers; Alanoly Andrews Subject: Re: [HACKERS] HS/SR on AIX On Thu, Aug 26, 2010 at 12:45 AM, Steve Singer <ssinger@ca.afilias.info> wrote: > A clean build from the beta4 source tarball where I'm careful to install > into a clean (ie no old beta2 artifacts laying around waiting to be > overwritten) isn't reproducing the issue. > > I'm happy to try other things if people suggest them (or if the original > reporter is still getting this after making sure he cleans up old files > first) but I'm thinking that was the issue. Thanks for the report! Alanoly, could you do a clean install and try the test again? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center **************************************************** This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Anydistribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized.If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courriel est confidentiel et prot�g�. L'exp�diteur ne renonce pas aux droits et obligations qui s'y rapportent. Toutediffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les)destinataire(s) d�sign�(s) est interdite. Si vous recevez ce courriel par erreur, veuillez m'en aviser imm�diatement,par retour de courriel ou par un autre moyen. ****************************************************
Alanoly Andrews wrote: > Fujii, > > All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installedin different and unique directories. > > Regards. Alanoly, If you do an export LIBPATH=/apps/pg_9.0_b4/lib before starting postgres on the replica does it make a difference? How about with a LIBPATH=/apps/pg_9.0_b4/lib/postgresql (I'm not exactly sure where libpq.a is on your install) > > Alanoly. -- Steve Singer Afilias Canada Data Services Developer 416-673-1142
Steve, I have tried all the LIBPATH's that you suggested. Besides, I don't think the problem is that postgres cannot find the "libpqwalreceiver"library. It does find it, but crashes on loading it. See below a repeat of the copy and paste from my firstpost, showing the sequence just before the crash: (dbx) where _alloc_initial_pthread(??) at 0x90000000049567c __pth_init(??) at 0x900000000493ba4 uload(??, ??, ??, ??, ??, ??, ??, ??) at 0x9fffffff0001954 load_64.load(??, ??, ??) at 0x90000000004686c loadAndInit() at 0x90000000047bd7c dlopen(??, ??) at 0x90000000011cc4c internal_load_library(libname = "/apps/pg_9.0_b4/lib/postgresql/libpqwalreceiver.so"), line 234 in "dfmgr.c" load_file(filename = "libpqwalreceiver", restricted = '\0'), line 156 in "dfmgr.c" WalReceiverMain(), line 248 in "walreceiver.c" AuxiliaryProcessMain(argc = 2, argv = 0x0fffffffffffa8b8), line 428 in "bootstrap.c" StartChildProcess(type = WalReceiverProcess), line 4405 in "postmaster.c" sigusr1_handler(postgres_signal_arg = 30), line 4227 in "postmaster.c" __fd_select(??, ??, ??, ??, ??) at 0x90000000011805c postmaster.select(__fds = 5, __readlist = 0x0fffffffffffd0a8, __writelist = (nil), __exceptlist = (nil), __timeout = 0x0ffffffffffff0c0),line 229 in "time.h" unnamed block in ServerLoop(), line 1391 in "postmaster.c" unnamed block in ServerLoop(), line 1391 in "postmaster.c" ServerLoop(), line 1391 in "postmaster.c" PostmasterMain(argc = 1, argv = 0x00000001102aa4b0), line 1092 in "postmaster.c" main(argc = 1, argv = 0x00000001102aa4b0), line 188 in "main.c" Alanoly. -----Original Message----- From: Steve Singer [mailto:ssinger@ca.afilias.info] Sent: Friday, August 27, 2010 3:45 PM To: Alanoly Andrews Cc: 'Fujii Masao'; Tom Lane; PostgreSQL-development Hackers; pgsql-admin@postgresql.org Subject: Re: [HACKERS] HS/SR on AIX Alanoly Andrews wrote: > Fujii, > > All my tests so far were done on "clean" installs. Every version I tested on, beta2 through beta4, was compiled and installedin different and unique directories. > > Regards. Alanoly, If you do an export LIBPATH=/apps/pg_9.0_b4/lib before starting postgres on the replica does it make a difference? How about with a LIBPATH=/apps/pg_9.0_b4/lib/postgresql (I'm not exactly sure where libpq.a is on your install) > > Alanoly. -- Steve Singer Afilias Canada Data Services Developer 416-673-1142 **************************************************** This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Anydistribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized.If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courriel est confidentiel et prot�g�. L'exp�diteur ne renonce pas aux droits et obligations qui s'y rapportent. Toutediffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les)destinataire(s) d�sign�(s) est interdite. Si vous recevez ce courriel par erreur, veuillez m'en aviser imm�diatement,par retour de courriel ou par un autre moyen. ****************************************************
Alanoly Andrews <alanolya@invera.com> writes:
> I have tried all the LIBPATH's that you suggested. Besides, I don't
> think the problem is that postgres cannot find the "libpqwalreceiver"
> library. It does find it, but crashes on loading it.
I thought the point of Steve's remarks was that it might be loading
the wrong version, either of libpqwalreceiver.so itself or libpq.so.
regards, tom lane