Thread: Streaming replication, loose ends
I've now committed streaming replication. I moved the files around a bit, and put the walreceiver/walsender stuff in a new src/backend/replication subdirectory. There's enough stuff there already to deserve a new subdirectory, and if we add the capability for streaming base backups etc. that has been discussed, we will have more code in there. But it's not time to party yet. There's still a few loose ends we need to address: Documentation. The patch originally moved around some sections, but I didn't include that in the committed version, to make it clear in the diff what exactly was added/changed. But I do agree with the original thought of adding a new "Replication" chapter, and moving all the replication and standby related stuff there from the "Backup and Restore" chapter, so let's start working on that. And of course the documentation needs to be improved and expanded in general. We talked about changing the retry-logic yesterday, so that the standby could fall back to restoring WAL files from archive even after it has already connected to the primary, if it e.g falls behind too much. It looks like that involves some heavy refactoring around ReadRecord/FetchRecord, which makes me a bit nervous given how critical ReadRecord() is and how old and well-tested the current code is. So let's tackle that as a follow-on patch. Then there's the issue of what privileges to require for a replication connection. I kept the superuser() check for now, so you currently need to be superuser, but as I opined earlier I don't think that's good for overall security. Perhaps we should add a new "replication" privilege besides the login privilege. To connect for replication, replication privilege would be checked instead of login privilege. That would make it quite simple to create a user or users for replication purposes, with no other access to the system. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas wrote: > I've now committed streaming replication. I moved the files around a > bit, and put the walreceiver/walsender stuff in a new > src/backend/replication subdirectory. There's enough stuff there already > to deserve a new subdirectory, and if we add the capability for > streaming base backups etc. that has been discussed, we will have more > code in there. > > But it's not time to party yet. There's still a few loose ends we need > to address: > > Documentation. The patch originally moved around some sections, but I > didn't include that in the committed version, to make it clear in the > diff what exactly was added/changed. But I do agree with the original > thought of adding a new "Replication" chapter, and moving all the > replication and standby related stuff there from the "Backup and > Restore" chapter, so let's start working on that. Uh, do we really want to call this "replication" rather than archive log streaming or something. It seems "replication" is a generic term and will confuse people who are using other replication solutions like Slony. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
2010/1/15 Bruce Momjian <bruce@momjian.us>: > Heikki Linnakangas wrote: >> I've now committed streaming replication. I moved the files around a >> bit, and put the walreceiver/walsender stuff in a new >> src/backend/replication subdirectory. There's enough stuff there already >> to deserve a new subdirectory, and if we add the capability for >> streaming base backups etc. that has been discussed, we will have more >> code in there. >> >> But it's not time to party yet. There's still a few loose ends we need >> to address: >> >> Documentation. The patch originally moved around some sections, but I >> didn't include that in the committed version, to make it clear in the >> diff what exactly was added/changed. But I do agree with the original >> thought of adding a new "Replication" chapter, and moving all the >> replication and standby related stuff there from the "Backup and >> Restore" chapter, so let's start working on that. > > Uh, do we really want to call this "replication" rather than archive log > streaming or something. It seems "replication" is a generic term and > will confuse people who are using other replication solutions like > Slony. +1 Pavel > > -- > Bruce Momjian <bruce@momjian.us> http://momjian.us > EnterpriseDB http://enterprisedb.com > > + If your life is a hard drive, Christ can be your backup. + > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
Hi, Heikki Linnakangas írta: > I've now committed streaming replication. I moved the files around a > bit, and put the walreceiver/walsender stuff in a new > src/backend/replication subdirectory. [snip] there's one loose end indeed. "make maintainer-clean" doesn't delete these: src/backend/replication/walreceiver/walreceiver.o src/backend/replication/walreceiver/walreceiver.so src/backend/replication/walreceiver/.deps/ Best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: "But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil." (Matthew 5:37) - basics of digital technology. "May your kingdom come" - superficial description of plate tectonics ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH http://www.postgresql.at/
Boszormenyi Zoltan wrote: > Heikki Linnakangas írta: >> I've now committed streaming replication. I moved the files around a >> bit, and put the walreceiver/walsender stuff in a new >> src/backend/replication subdirectory. [snip] > > there's one loose end indeed. > "make maintainer-clean" doesn't delete these: > > src/backend/replication/walreceiver/walreceiver.o > src/backend/replication/walreceiver/walreceiver.so > src/backend/replication/walreceiver/.deps/ Hmm, I think I'm going to need some help with the Makefiles. Clearly the way I hooked that directory to the build system was wrong. I think the attached patch will fix that, but I wonder if there's a trick I'm missing. Do the MSVC scripts need adjusting? 'red_bat' compiled fine, so I guess not. Also, I'm seeing a failure in buildfarm member 'colugos': /opt/local/bin/ccache /Developer/usr/bin/llvm-gcc-4.2 -no-cpp-precomp -I/opt/local/include -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -g -bundle -multiply_defined suppress walreceiver.o -bundle_loader ../../../../src/backend/postgres -L../../../../src/interfaces/libpq -L../../../../src/port -L/opt/local/lib -lpq -o walreceiver.so ld: library not found for -lpq collect2: ld returned 1 exit status make[2]: *** [walreceiver.so] Error 1 make[2]: *** Waiting for unfinished jobs.... I suspect that's because libpq isn't built yet. I have this: > all: submake-libpq all-shared-lib in src/backend/replication/walreceiver/Makefile, but is that not enough? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ? GNUmakefile ? b ? config.log ? config.status ? config.status.lineno ? configure.lineno ? gin-splay-1.patch ? gin-splay-2.patch ? gin-splay-3.patch ? md-1.c ? md-1.patch ? temp-file-resowner-2.patch ? contrib/pgbench/fsynctest ? contrib/pgbench/fsynctest.c ? contrib/pgbench/fsynctestfile ? contrib/spi/.deps ? src/Makefile.global ? src/backend/aaa.patch ? src/backend/postgres ? src/backend/access/common/.deps ? src/backend/access/gin/.deps ? src/backend/access/gist/.deps ? src/backend/access/hash/.deps ? src/backend/access/heap/.deps ? src/backend/access/index/.deps ? src/backend/access/nbtree/.deps ? src/backend/access/transam/.deps ? src/backend/bootstrap/.deps ? src/backend/catalog/.deps ? src/backend/commands/.deps ? src/backend/executor/.deps ? src/backend/foreign/.deps ? src/backend/foreign/dummy/.deps ? src/backend/foreign/postgresql/.deps ? src/backend/lib/.deps ? src/backend/libpq/.deps ? src/backend/main/.deps ? src/backend/nodes/.deps ? src/backend/optimizer/geqo/.deps ? src/backend/optimizer/path/.deps ? src/backend/optimizer/plan/.deps ? src/backend/optimizer/prep/.deps ? src/backend/optimizer/util/.deps ? src/backend/parser/.deps ? src/backend/po/af.mo ? src/backend/po/cs.mo ? src/backend/po/hr.mo ? src/backend/po/hu.mo ? src/backend/po/it.mo ? src/backend/po/ko.mo ? src/backend/po/nb.mo ? src/backend/po/nl.mo ? src/backend/po/pl.mo ? src/backend/po/ro.mo ? src/backend/po/ru.mo ? src/backend/po/sk.mo ? src/backend/po/sl.mo ? src/backend/po/sv.mo ? src/backend/po/zh_CN.mo ? src/backend/po/zh_TW.mo ? src/backend/port/.deps ? src/backend/postmaster/.deps ? src/backend/regex/.deps ? src/backend/replication/.deps ? src/backend/replication/walreceiver/.deps ? src/backend/rewrite/.deps ? src/backend/snowball/.deps ? src/backend/snowball/snowball_create.sql ? src/backend/storage/buffer/.deps ? src/backend/storage/file/.deps ? src/backend/storage/freespace/.deps ? src/backend/storage/ipc/.deps ? src/backend/storage/large_object/.deps ? src/backend/storage/lmgr/.deps ? src/backend/storage/page/.deps ? src/backend/storage/smgr/.deps ? src/backend/tcop/.deps ? src/backend/tsearch/.deps ? src/backend/utils/.deps ? src/backend/utils/probes.h ? src/backend/utils/adt/.deps ? src/backend/utils/cache/.deps ? src/backend/utils/error/.deps ? src/backend/utils/fmgr/.deps ? src/backend/utils/hash/.deps ? src/backend/utils/init/.deps ? src/backend/utils/mb/.deps ? src/backend/utils/mb/Unicode/BIG5.TXT ? src/backend/utils/mb/Unicode/CP950.TXT ? src/backend/utils/mb/conversion_procs/conversion_create.sql ? src/backend/utils/mb/conversion_procs/ascii_and_mic/.deps ? src/backend/utils/mb/conversion_procs/cyrillic_and_mic/.deps ? src/backend/utils/mb/conversion_procs/euc2004_sjis2004/.deps ? src/backend/utils/mb/conversion_procs/euc_cn_and_mic/.deps ? src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/.deps ? src/backend/utils/mb/conversion_procs/euc_jp_and_sjis/.deps ? src/backend/utils/mb/conversion_procs/euc_kr_and_mic/.deps ? src/backend/utils/mb/conversion_procs/euc_tw_and_big5/.deps ? src/backend/utils/mb/conversion_procs/latin2_and_win1250/.deps ? src/backend/utils/mb/conversion_procs/latin_and_mic/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_ascii/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_big5/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_cyrillic/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc2004/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc_cn/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc_jis_2004/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc_jp/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc_kr/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_euc_tw/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_gb18030/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_gbk/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_iso8859/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_johab/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_shift_jis_2004/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_sjis/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_sjis2004/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_uhc/.deps ? src/backend/utils/mb/conversion_procs/utf8_and_win/.deps ? src/backend/utils/misc/.deps ? src/backend/utils/mmgr/.deps ? src/backend/utils/resowner/.deps ? src/backend/utils/sort/.deps ? src/backend/utils/time/.deps ? src/bin/initdb/.deps ? src/bin/initdb/initdb ? src/bin/initdb/po/ko.mo ? src/bin/initdb/po/pl.mo ? src/bin/initdb/po/ro.mo ? src/bin/initdb/po/sk.mo ? src/bin/initdb/po/sl.mo ? src/bin/initdb/po/zh_CN.mo ? src/bin/initdb/po/zh_TW.mo ? src/bin/pg_config/.deps ? src/bin/pg_config/pg_config ? src/bin/pg_config/po/cs.mo ? src/bin/pg_config/po/pl.mo ? src/bin/pg_config/po/sl.mo ? src/bin/pg_config/po/zh_CN.mo ? src/bin/pg_config/po/zh_TW.mo ? src/bin/pg_controldata/.deps ? src/bin/pg_controldata/pg_controldata ? src/bin/pg_controldata/po/cs.mo ? src/bin/pg_controldata/po/fa.mo ? src/bin/pg_controldata/po/hu.mo ? src/bin/pg_controldata/po/nb.mo ? src/bin/pg_controldata/po/pl.mo ? src/bin/pg_controldata/po/ro.mo ? src/bin/pg_controldata/po/ru.mo ? src/bin/pg_controldata/po/sk.mo ? src/bin/pg_controldata/po/sl.mo ? src/bin/pg_controldata/po/zh_CN.mo ? src/bin/pg_controldata/po/zh_TW.mo ? src/bin/pg_ctl/.deps ? src/bin/pg_ctl/pg_ctl ? src/bin/pg_ctl/po/cs.mo ? src/bin/pg_ctl/po/ro.mo ? src/bin/pg_ctl/po/sk.mo ? src/bin/pg_ctl/po/sl.mo ? src/bin/pg_ctl/po/zh_CN.mo ? src/bin/pg_ctl/po/zh_TW.mo ? src/bin/pg_dump/.deps ? src/bin/pg_dump/pg_dump ? src/bin/pg_dump/pg_dumpall ? src/bin/pg_dump/pg_restore ? src/bin/pg_dump/po/cs.mo ? src/bin/pg_dump/po/ko.mo ? src/bin/pg_dump/po/nb.mo ? src/bin/pg_dump/po/ro.mo ? src/bin/pg_dump/po/ru.mo ? src/bin/pg_dump/po/sk.mo ? src/bin/pg_dump/po/sl.mo ? src/bin/pg_dump/po/zh_CN.mo ? src/bin/pg_dump/po/zh_TW.mo ? src/bin/pg_resetxlog/.deps ? src/bin/pg_resetxlog/pg_resetxlog ? src/bin/pg_resetxlog/po/cs.mo ? src/bin/pg_resetxlog/po/hu.mo ? src/bin/pg_resetxlog/po/nb.mo ? src/bin/pg_resetxlog/po/sk.mo ? src/bin/pg_resetxlog/po/sl.mo ? src/bin/pg_resetxlog/po/zh_CN.mo ? src/bin/pg_resetxlog/po/zh_TW.mo ? src/bin/psql/.deps ? src/bin/psql/psql ? src/bin/psql/po/fa.mo ? src/bin/psql/po/hu.mo ? src/bin/psql/po/it.mo ? src/bin/psql/po/ko.mo ? src/bin/psql/po/nb.mo ? src/bin/psql/po/ro.mo ? src/bin/psql/po/ru.mo ? src/bin/psql/po/sk.mo ? src/bin/psql/po/sl.mo ? src/bin/psql/po/zh_CN.mo ? src/bin/psql/po/zh_TW.mo ? src/bin/scripts/.deps ? src/bin/scripts/clusterdb ? src/bin/scripts/createdb ? src/bin/scripts/createlang ? src/bin/scripts/createuser ? src/bin/scripts/dropdb ? src/bin/scripts/droplang ? src/bin/scripts/dropuser ? src/bin/scripts/reindexdb ? src/bin/scripts/vacuumdb ? src/bin/scripts/po/ru.mo ? src/bin/scripts/po/sk.mo ? src/bin/scripts/po/sl.mo ? src/bin/scripts/po/zh_CN.mo ? src/bin/scripts/po/zh_TW.mo ? src/include/pg_config.h ? src/include/stamp-h ? src/interfaces/ecpg/compatlib/.deps ? src/interfaces/ecpg/compatlib/exports.list ? src/interfaces/ecpg/compatlib/libecpg_compat.so.3.1 ? src/interfaces/ecpg/compatlib/libecpg_compat.so.3.2 ? src/interfaces/ecpg/ecpglib/.deps ? src/interfaces/ecpg/ecpglib/exports.list ? src/interfaces/ecpg/ecpglib/libecpg.so.6.1 ? src/interfaces/ecpg/ecpglib/libecpg.so.6.2 ? src/interfaces/ecpg/include/ecpg_config.h ? src/interfaces/ecpg/include/stamp-h ? src/interfaces/ecpg/pgtypeslib/.deps ? src/interfaces/ecpg/pgtypeslib/exports.list ? src/interfaces/ecpg/pgtypeslib/libpgtypes.so.3.1 ? src/interfaces/ecpg/pgtypeslib/libpgtypes.so.3.2 ? src/interfaces/ecpg/preproc/.deps ? src/interfaces/ecpg/preproc/ecpg ? src/interfaces/libpq/.deps ? src/interfaces/libpq/exports.list ? src/interfaces/libpq/libpq.so.5.2 ? src/interfaces/libpq/libpq.so.5.3 ? src/interfaces/libpq/po/af.mo ? src/interfaces/libpq/po/hr.mo ? src/interfaces/libpq/po/nb.mo ? src/interfaces/libpq/po/pl.mo ? src/interfaces/libpq/po/sk.mo ? src/interfaces/libpq/po/sl.mo ? src/interfaces/libpq/po/zh_CN.mo ? src/interfaces/libpq/po/zh_TW.mo ? src/pl/plpgsql/src/.deps ? src/pl/plpgsql/src/pl_scan.c ? src/pl/plpgsql/src/po/tr.mo ? src/port/.deps ? src/port/pg_config_paths.h ? src/test/regress/.deps ? src/test/regress/pg_regress ? src/test/regress/testtablespace ? src/timezone/.deps ? src/timezone/zic Index: src/Makefile =================================================================== RCS file: /cvsroot/pgsql/src/Makefile,v retrieving revision 1.48 diff -u -r1.48 Makefile --- src/Makefile 15 Jan 2010 09:18:59 -0000 1.48 +++ src/Makefile 15 Jan 2010 12:25:57 -0000 @@ -18,7 +18,6 @@ $(MAKE) -C timezone $@ $(MAKE) -C backend $@ $(MAKE) -C backend/utils/mb/conversion_procs $@ - $(MAKE) -C backend/replication/walreceiver $@ $(MAKE) -C backend/snowball $@ $(MAKE) -C include $@ $(MAKE) -C interfaces $@ Index: src/backend/replication/Makefile =================================================================== RCS file: /cvsroot/pgsql/src/backend/replication/Makefile,v retrieving revision 1.1 diff -u -r1.1 Makefile --- src/backend/replication/Makefile 15 Jan 2010 09:19:03 -0000 1.1 +++ src/backend/replication/Makefile 15 Jan 2010 12:25:58 -0000 @@ -15,3 +15,9 @@ OBJS = walsender.o walreceiverfuncs.o include $(top_srcdir)/src/backend/common.mk + +# We can't include walreceiver subdiretory using SUBDIR, because that would +# link it into the main 'postgres' binary. We want it to be compiled into +# a separate shared object. +all install installdirs uninstall distprep clean distclean maintainer-clean: + $(MAKE) -C walreceiver $@
Heikki Linnakangas wrote: > Also, I'm seeing a failure in buildfarm member 'colugos': > > /opt/local/bin/ccache /Developer/usr/bin/llvm-gcc-4.2 -no-cpp-precomp > -I/opt/local/include -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing > -fwrapv -g -bundle -multiply_defined suppress walreceiver.o > -bundle_loader ../../../../src/backend/postgres > -L../../../../src/interfaces/libpq -L../../../../src/port > -L/opt/local/lib -lpq -o walreceiver.so > ld: library not found for -lpq > collect2: ld returned 1 exit status > make[2]: *** [walreceiver.so] Error 1 > make[2]: *** Waiting for unfinished jobs.... > > I suspect that's because libpq isn't built yet. I have this: > >> all: submake-libpq all-shared-lib > > in src/backend/replication/walreceiver/Makefile, but is that not enough? Yep. What's happening is that "make -j" starts building libpq and walreceiver.so simultaneously, because of the above line in the Makefile. We actually have the same problem in src/bin/*/Makefile, but we don't notice it there because src/interfaces is listed before src/bin in src/Makefile, so when you do "make -j" at the top-level, libpq is built first. You get the same error if you do "make clean" at the top-level, and then e.g "cd src/bin/scripts/; make -j" So the simple fix would be to reorder the lines in src/Makefile, so that src/interfaces is built before src/backend. Alternatively we could do this: *** src/backend/replication/walreceiver/Makefile 15 Jan 2010 09:19:03 -0000 1.1 --- src/backend/replication/walreceiver/Makefile 15 Jan 2010 13:57:24 -0000 *************** *** 18,24 **** SHLIB_LINK = $(libpq) NAME = walreceiver ! all: submake-libpq all-shared-lib include $(top_srcdir)/src/Makefile.shlib --- 18,28 ---- SHLIB_LINK = $(libpq) NAME = walreceiver ! all: all-shared-lib ! ! # Compiling walreceiver.o doesn't really need libpq library, ! # only linking it does. But there's no easy way to specify that. ! walreceiver.o: submake-libpq include $(top_srcdir)/src/Makefile.shlib And I guess all the other uses of submake-libpq should be changed similarly. Am I missing a trick? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> Yep. What's happening is that "make -j" starts building libpq and
> walreceiver.so simultaneously, because of the above line in the
> Makefile. We actually have the same problem in src/bin/*/Makefile, but
> we don't notice it there because src/interfaces is listed before src/bin
> in src/Makefile, so when you do "make -j" at the top-level, libpq is
> built first.
I'm actually fairly uncomfortable with the notion that something buried
deep within the src/backend tree is going to reach over and cause libpq
to get built. Maybe the real answer is that you put walreceiver in the
wrong place, and it ought to be under src/bin/.
regards, tom lane
Tom Lane wrote: > Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes: >> Yep. What's happening is that "make -j" starts building libpq and >> walreceiver.so simultaneously, because of the above line in the >> Makefile. We actually have the same problem in src/bin/*/Makefile, but >> we don't notice it there because src/interfaces is listed before src/bin >> in src/Makefile, so when you do "make -j" at the top-level, libpq is >> built first. > > I'm actually fairly uncomfortable with the notion that something buried > deep within the src/backend tree is going to reach over and cause libpq > to get built. Maybe the real answer is that you put walreceiver in the > wrong place, and it ought to be under src/bin/. That feels even more wrong to me. Walreceiver is a postmaster subprocess, tightly integrated with the rest of the backend. One can argue that it shouldn't be, and walreceiver process should call libpq through some new API, and the builtin implementation of that API which uses libpq would be a loadable module that could be in src/bin/ or contrib. Greg Stark requested that earlier. But I don't want to start designing such an API at this point. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Fri, Jan 15, 2010 at 11:47 AM, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote: > Tom Lane wrote: >> Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes: >>> Yep. What's happening is that "make -j" starts building libpq and >>> walreceiver.so simultaneously, because of the above line in the >>> Makefile. We actually have the same problem in src/bin/*/Makefile, but >>> we don't notice it there because src/interfaces is listed before src/bin >>> in src/Makefile, so when you do "make -j" at the top-level, libpq is >>> built first. >> >> I'm actually fairly uncomfortable with the notion that something buried >> deep within the src/backend tree is going to reach over and cause libpq >> to get built. Maybe the real answer is that you put walreceiver in the >> wrong place, and it ought to be under src/bin/. > > That feels even more wrong to me. Walreceiver is a postmaster > subprocess, tightly integrated with the rest of the backend. The major problem with having one part of the tree depend on a completely different part of the tree is that it's easy for the dependencies to be wrong. If the backend depends on libpq, then it depends implicitly on all the things on which libpq depends. If something that libpq depends on, but that the backend does not depend on directly, gets updated, does the backend get rebuilt? It's easy to get this wrong. On the other hand, it's also possible to get it right. If we can decide what we want to happen, I'm willing to take a crack at it, though if you or Tom or Peter prefer to do it that is certainly OK with me too. ...Robert
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> Tom Lane wrote:
>> I'm actually fairly uncomfortable with the notion that something buried
>> deep within the src/backend tree is going to reach over and cause libpq
>> to get built. Maybe the real answer is that you put walreceiver in the
>> wrong place, and it ought to be under src/bin/.
> That feels even more wrong to me. Walreceiver is a postmaster
> subprocess, tightly integrated with the rest of the backend.
[ shrug... ] pg_dump, to take one example, is considerably more
"tightly integrated" with the backend than walreceiver is.
regards, tom lane
Robert Haas wrote: > On Fri, Jan 15, 2010 at 11:47 AM, Heikki Linnakangas > <heikki.linnakangas@enterprisedb.com> wrote: >> Tom Lane wrote: >>> Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes: >>>> Yep. What's happening is that "make -j" starts building libpq and >>>> walreceiver.so simultaneously, because of the above line in the >>>> Makefile. We actually have the same problem in src/bin/*/Makefile, but >>>> we don't notice it there because src/interfaces is listed before src/bin >>>> in src/Makefile, so when you do "make -j" at the top-level, libpq is >>>> built first. >>> I'm actually fairly uncomfortable with the notion that something buried >>> deep within the src/backend tree is going to reach over and cause libpq >>> to get built. Maybe the real answer is that you put walreceiver in the >>> wrong place, and it ought to be under src/bin/. >> That feels even more wrong to me. Walreceiver is a postmaster >> subprocess, tightly integrated with the rest of the backend. > > The major problem with having one part of the tree depend on a > completely different part of the tree is that it's easy for the > dependencies to be wrong. If the backend depends on libpq, then it > depends implicitly on all the things on which libpq depends. If > something that libpq depends on, but that the backend does not depend > on directly, gets updated, does the backend get rebuilt? The backend doesn't get rebuilt, and it doesn't need to be. The fact that walreceiver is a dynamically loaded module should isolate changes in libpq or its dependencies from affecting the rest of the backend. I moved the line for src/backend/replication/walreceiver in src/Makefile further down, after src/interfaces. That should fix the build failures for now, but I'm all ears if there's better suggestions. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> I moved the line for src/backend/replication/walreceiver in src/Makefile
> further down, after src/interfaces. That should fix the build failures
> for now, but I'm all ears if there's better suggestions.
Yeah, I saw. Seems like a reasonable solution for now.
regards, tom lane
On Fri, 2010-01-15 at 05:36 -0500, Bruce Momjian wrote: > Heikki Linnakangas wrote: > > I've now committed streaming replication. I moved the files around a > > bit, and put the walreceiver/walsender stuff in a new > > src/backend/replication subdirectory. There's enough stuff there already > > to deserve a new subdirectory, and if we add the capability for > > streaming base backups etc. that has been discussed, we will have more > > code in there. > > > > But it's not time to party yet. There's still a few loose ends we need > > to address: > > > > Documentation. The patch originally moved around some sections, but I > > didn't include that in the committed version, to make it clear in the > > diff what exactly was added/changed. But I do agree with the original > > thought of adding a new "Replication" chapter, and moving all the > > replication and standby related stuff there from the "Backup and > > Restore" chapter, so let's start working on that. > > Uh, do we really want to call this "replication" rather than archive log > streaming or something. It seems "replication" is a generic term and > will confuse people who are using other replication solutions like > Slony. +1, it is not replication. I would call it something like continuous archiving or streaming pitr Joshua D. Drake > > -- > Bruce Momjian <bruce@momjian.us> http://momjian.us > EnterpriseDB http://enterprisedb.com > > + If your life is a hard drive, Christ can be your backup. + > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, Jan 15, 2010 at 12:17 PM, Joshua D. Drake <jd@commandprompt.com> wrote: >> Uh, do we really want to call this "replication" rather than archive log >> streaming or something. It seems "replication" is a generic term and >> will confuse people who are using other replication solutions like >> Slony. > > +1, it is not replication. I would call it something like continuous > archiving or streaming pitr Of course "PITR" does stand for "point-in-time replication"... ...Robert
Bruce Momjian wrote: > Heikki Linnakangas wrote: >> I've now committed streaming replication. I moved the files around a >> bit, and put the walreceiver/walsender stuff in a new >> src/backend/replication subdirectory. There's enough stuff there already >> to deserve a new subdirectory, and if we add the capability for >> streaming base backups etc. that has been discussed, we will have more >> code in there. >> >> But it's not time to party yet. There's still a few loose ends we need >> to address: >> >> Documentation. The patch originally moved around some sections, but I >> didn't include that in the committed version, to make it clear in the >> diff what exactly was added/changed. But I do agree with the original >> thought of adding a new "Replication" chapter, and moving all the >> replication and standby related stuff there from the "Backup and >> Restore" chapter, so let's start working on that. > > Uh, do we really want to call this "replication" rather than archive log > streaming or something. It seems "replication" is a generic term and > will confuse people who are using other replication solutions like > Slony. Good question. OTOH, if we move the sections about setting up a file-shipping based standby with pg_standby, that's not streaming. What we have now is: Server Administration ... Backup and Restore SQL Dump File System Level Backup Continuous Archiving and Point-In-TimeRecovery (PITR) Warm Standby Servers for High Availability Planning Implementation Failover Record-based Log Shipping Streaming Replication Incrementally Updated Backups Hot Standby User's Overview Handling query conflicts Administrator's Overview Hot Standby Parameter Reference Caveats High Availability,Load Balancing, and Replication I propose: Server Administration Backup and Restore SQL Dump File System Level Backup Continuous Archiving and Point-In-Time Recovery(PITR) High Availability, Load Balancing, and Replication Introduction Comparison of different solutions (*) File-based Log Shipping Planning Implementation Streaming Replication Setting up Failover Hot Standby User's Overview Handling query conflicts Administrator's Overview Hot Standby Parameter Reference Caveats Incrementally Updated Backups (*) Current content of "High Availability, Load Balancing, and Replication" chapter goes here Note that I propose to remove "Record-based Log Shipping" section altogether. We can briefly mention that method under Streaming Replication, but I consider that obsolete with streaming replication. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Robert Haas wrote: > On Fri, Jan 15, 2010 at 12:17 PM, Joshua D. Drake <jd@commandprompt.com> wrote: >>> Uh, do we really want to call this "replication" rather than archive log >>> streaming or something. It seems "replication" is a generic term and >>> will confuse people who are using other replication solutions like >>> Slony. >> +1, it is not replication. I would call it something like continuous >> archiving or streaming pitr > > Of course "PITR" does stand for "point-in-time replication"... I'm not sure if you're joking, but PITR actually stands for Point-In-Time *Recovery*. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Fri, 2010-01-15 at 19:30 +0200, Heikki Linnakangas wrote: > Robert Haas wrote: > > On Fri, Jan 15, 2010 at 12:17 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > >>> Uh, do we really want to call this "replication" rather than archive log > >>> streaming or something. It seems "replication" is a generic term and > >>> will confuse people who are using other replication solutions like > >>> Slony. > >> +1, it is not replication. I would call it something like continuous > >> archiving or streaming pitr > > > > Of course "PITR" does stand for "point-in-time replication"... > > I'm not sure if you're joking, but PITR actually stands for > Point-In-Time *Recovery*. Right. Joshua D. Drake > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, Jan 15, 2010 at 12:30 PM, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote: > Robert Haas wrote: >> On Fri, Jan 15, 2010 at 12:17 PM, Joshua D. Drake <jd@commandprompt.com> wrote: >>>> Uh, do we really want to call this "replication" rather than archive log >>>> streaming or something. It seems "replication" is a generic term and >>>> will confuse people who are using other replication solutions like >>>> Slony. >>> +1, it is not replication. I would call it something like continuous >>> archiving or streaming pitr >> >> Of course "PITR" does stand for "point-in-time replication"... > > I'm not sure if you're joking, but PITR actually stands for > Point-In-Time *Recovery*. Oops. No, not joking, just wrong. But I'm still wondering why this isn't replication. ...Robert
On Jan 15, 2010, at 9:37 AM, Robert Haas wrote: > But I'm still wondering why this isn't replication. I was wondering the same thing. ISTM that the docs could reference third-party replication solutions, too (or a wiki pagelisting them, since they'll change often). Anyway, I think Heikki's proposed chapter name covers it: > High Availability, Load Balancing, and Replication Works for me. Best, David
On Fri, Jan 15, 2010 at 12:11:01PM -0500, Tom Lane wrote: > Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes: > > I moved the line for src/backend/replication/walreceiver in > > src/Makefile further down, after src/interfaces. That should fix > > the build failures for now, but I'm all ears if there's better > > suggestions. > > Yeah, I saw. Seems like a reasonable solution for now. We can always cvs mv...oh, wait! ;) Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 >>>> +1, it is not replication. I would call it something like continuous >>>> archiving or streaming pitr I agree we should consider a different name. > But I'm still wondering why this isn't replication. Because replication is an ambiguous and overloaded term. On the other hand, it's a great buzzword, so we should use the phrase as much as possible in the press releases. :) - -- Greg Sabino Mullane greg@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201001151257 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktQrNYACgkQvJuQZxSWSsj8NACgwIQVq/GGQoY/4U6pAYyR5IeS EsoAoLjoYlLErv4g3Vy65rbA9u9W0vww =u2Hk -----END PGP SIGNATURE-----
On Fri, 2010-01-15 at 19:29 +0200, Heikki Linnakangas wrote: > File-based Log Shipping > Planning > Implementation > Streaming Replication > Setting up How about "Log Streaming Replication"? So its a particular kind of replication, which seems correct to me. -- Simon Riggs www.2ndQuadrant.com
On Fri, Jan 15, 2010 at 6:39 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > On Fri, 2010-01-15 at 19:29 +0200, Heikki Linnakangas wrote: > How about "Log Streaming Replication"? > > So its a particular kind of replication, which seems correct to me. I thought the whole point of this effort was to be able to bill it as a built-in easy replication which was perceived as a major hole in Postgres's feature set. -- greg
On Fri, 2010-01-15 at 22:38 +0000, Greg Stark wrote: > On Fri, Jan 15, 2010 at 6:39 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > > On Fri, 2010-01-15 at 19:29 +0200, Heikki Linnakangas wrote: > > How about "Log Streaming Replication"? > > > > So its a particular kind of replication, which seems correct to me. > > I thought the whole point of this effort was to be able to bill it as > a built-in easy replication which was perceived as a major hole in > Postgres's feature set. How does that affect my proposal? Heikki called it Streaming Replication; I have just added "Log" to it, to make clear that it is similar in many ways to File-based Log Shipping, which was a section heading earlier in docs. -- Simon Riggs www.2ndQuadrant.com
On Fri, 2010-01-15 at 05:36 -0500, Bruce Momjian wrote: > Heikki Linnakangas wrote: > > I've now committed streaming replication. I moved the files around a > > bit, and put the walreceiver/walsender stuff in a new > > src/backend/replication subdirectory. There's enough stuff there already > > to deserve a new subdirectory, and if we add the capability for > > streaming base backups etc. that has been discussed, we will have more > > code in there. > > > > But it's not time to party yet. There's still a few loose ends we need > > to address: > > > > Documentation. The patch originally moved around some sections, but I > > didn't include that in the committed version, to make it clear in the > > diff what exactly was added/changed. But I do agree with the original > > thought of adding a new "Replication" chapter, and moving all the > > replication and standby related stuff there from the "Backup and > > Restore" chapter, so let's start working on that. > > Uh, do we really want to call this "replication" rather than archive log > streaming or something. It seems "replication" is a generic term and > will confuse people who are using other replication solutions like > Slony. +1, it is not replication. I would call it something like continuous archiving or streaming pitr Joshua D. Drake > > -- > Bruce Momjian <bruce@momjian.us> http://momjian.us > EnterpriseDB http://enterprisedb.com > > + If your life is a hard drive, Christ can be your backup. + > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
On Fri, 2010-01-15 at 19:30 +0200, Heikki Linnakangas wrote: > Robert Haas wrote: > > On Fri, Jan 15, 2010 at 12:17 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > >>> Uh, do we really want to call this "replication" rather than archive log > >>> streaming or something. It seems "replication" is a generic term and > >>> will confuse people who are using other replication solutions like > >>> Slony. > >> +1, it is not replication. I would call it something like continuous > >> archiving or streaming pitr > > > > Of course "PITR" does stand for "point-in-time replication"... > > I'm not sure if you're joking, but PITR actually stands for > Point-In-Time *Recovery*. Right. Joshua D. Drake > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
Heikki Linnakangas wrote: > > Uh, do we really want to call this "replication" rather than archive log > > streaming or something. It seems "replication" is a generic term and > > will confuse people who are using other replication solutions like > > Slony. > > Good question. OTOH, if we move the sections about setting up a > file-shipping based standby with pg_standby, that's not streaming. > > What we have now is: Frankly, I am concerned we now have a "replication" CVS subdirectory; it looks more like a 'wal_streaming' directory to me. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Simon Riggs wrote: > On Fri, 2010-01-15 at 19:29 +0200, Heikki Linnakangas wrote: > > > File-based Log Shipping > > Planning > > Implementation > > Streaming Replication > > Setting up > > How about "Log Streaming Replication"? > > So its a particular kind of replication, which seems correct to me. Yea, I like that. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +