Thread: Replication direction
I was wondering what direction we want to take for replication? There is a group at: http://gborg.postgresql.org/genpage?replication_research working on a fairly advanced replication solution that they hope to add to 7.3. However, has everyone reviewed that information? Is this the direction we want to go? I was personally impressed by it, but I am no replication expert. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> working on a fairly advanced replication solution that they hope to add
> to 7.3. However, has everyone reviewed that information? Is this the
> direction we want to go? I was personally impressed by it, but I am no
> replication expert.
7.3 is probably a reach. We're hoping to get master-slave working first
and then multimaster, high quality multimaster for 7.3 might be too fast
to hope for (even it it takes a year as 7.1 -> 7.2 did).
- Brandon
----------------------------------------------------------------------------c: 646-456-5455
h: 201-798-4983b. palmer, bpalmer@crimelabs.net pgp:crimelabs.net/bpalmer.pgp5
One thought I had about replication is base it off of the WAL files. As they are created, they can be forward into other servers (both on a LAN and WAN environment) and imported to mirror the originating database. (As opposed to having triggers that force writes into replicated servers on updates, inserts and deletes) On Fri, 2002-02-22 at 08:57, Bruce Momjian wrote: > I was wondering what direction we want to take for replication? There > is a group at: > > http://gborg.postgresql.org/genpage?replication_research > > working on a fairly advanced replication solution that they hope to add > to 7.3. However, has everyone reviewed that information? Is this the > direction we want to go? I was personally impressed by it, but I am no > replication expert. > > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 853-3000 > + If your life is a hard drive, | 830 Blythe Avenue > + Christ can be your backup. | Drexel Hill, Pennsylvania 19026 > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html -- Virtually, Ned Wolpert <ned.wolpert@knowledgenet.com> D08C2F45: 28E7 56CB 58AC C622 5A51 3C42 8B2B 2739 D08C 2F45
bpalmer wrote: > > working on a fairly advanced replication solution that they hope to add > > to 7.3. However, has everyone reviewed that information? Is this the > > direction we want to go? I was personally impressed by it, but I am no > > replication expert. > > 7.3 is probably a reach. We're hoping to get master-slave working first > and then multimaster, high quality multimaster for 7.3 might be too fast > to hope for (even it it takes a year as 7.1 -> 7.2 did). Sorry, yes master-slave for 7.3. The idea is that the code will be there in 7.3 for master-slave, and we will add on that for 7.4 and later, but that clearly the infrastructure is there for master-master from the start. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Ned Wolpert wrote: > One thought I had about replication is base it off of the WAL files. As > they are created, they can be forward into other servers (both on a LAN > and WAN environment) and imported to mirror the originating database. > (As opposed to having triggers that force writes into replicated servers > on updates, inserts and deletes) OK, let me chime in here and give my observations, then I will address the WAL issue. First, in all the discussions I have heard, the final replication solution needs to be: multi-mastergood performancelow maintenance (no conflict resolution rules) Now, at the URL I mentioned, there are 5+ other pgsql replication solutions listed: http://gborg.postgresql.org/genpage?replication_research However, I don't think any of these meet this criteria, particularly because they are not integrated into the backend code. That doesn't mean they aren't valuable, but that they don't represent a final solution for pgsql replication. As far as I know, the only working project that will meet these needs is the one at that URL based on Postgres-R. What impressed me about it was that it did not use two-phase commit, nor conflict resolution rules, but rather used a queue of changes that gets broadcast to all the servers in the replication cluster. This seems to allow high throughput with low maintenance. The actual direction of the project was decided by researching all the current solutions and educational research to find the strengths and weaknesses of each one. As far as WAL, it currently has the tid's in the WAL file which will not match other servers unless those servers are read-only. It is tempting to think that WAL or some other existing mechanism will allow us to do replication cheaply, but it is my understanding that the interactions of multiple write server is quite complicated and requires an amount of overhead mechanism that is similar to our current transaction mechanisms that allow multiple people to modify the same table. People know I am all for the quick solution if it fits into our existing code, but I am afraid replication is one of those items that has to be designed from the ground up on a foundation that is backed by research and experts in the field. I think this replication project has the potential to give us a replication capability that is better than the commercial offering of other databases. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> One thought I had about replication is base it off of the WAL files. As
> they are created, they can be forward into other servers (both on a LAN
> and WAN environment) and imported to mirror the originating database.
> (As opposed to having triggers that force writes into replicated servers
> on updates, inserts and deletes)
In that case, we would have a master / slave environment. Also, how do
we guarantee that the slaves get the info and are able to commit it? Next
problem: a wal has all tuples changed. though "delete from foo" is only
a few characters, the WAL would be however many rows are in the database
(that were deleted).
That said, however, we are looking at using some of the same methods
that are used to generate the WAL info to use for packaging info to go to
other hosts in the network (for pg-repl).
- Brandon
----------------------------------------------------------------------------c: 646-456-5455
h: 201-798-4983b. palmer, bpalmer@crimelabs.net pgp:crimelabs.net/bpalmer.pgp5
On Fri, 2002-02-22 at 10:20, Bruce Momjian wrote: > As far as WAL, it currently has the tid's in the WAL file which will not > match other servers unless those servers are read-only. It is tempting > to think that WAL or some other existing mechanism will allow us to do > replication cheaply, but it is my understanding that the interactions of > multiple write server is quite complicated and requires an amount of > overhead mechanism that is similar to our current transaction mechanisms > that allow multiple people to modify the same table. Drat. I was hoping that the WAL was an overlooked 'easy' solution... but from the sound of this thread, the WAL misses some key issues. And it doesn't help the master-master environment at all, which I forgot. Ah well... > People know I am all for the quick solution if it fits into our existing > code, but I am afraid replication is one of those items that has to be > designed from the ground up on a foundation that is backed by research > and experts in the field. I think this replication project has the > potential to give us a replication capability that is better than > the commercial offering of other databases. -- Virtually, Ned Wolpert <ned.wolpert@knowledgenet.com> D08C2F45: 28E7 56CB 58AC C622 5A51 3C42 8B2B 2739 D08C 2F45
> The actual direction of the project was decided by researching all > the current solutions and educational research to find the strengths > and weaknesses of each one. For those that are interested, http://www.spread.org/ Spread toolkit is what's being used. -sc -- Sean Chittenden