Re: Merge a sharded master into a single read-only slave - Mailing list pgsql-general

From Keith Fiske
Subject Re: Merge a sharded master into a single read-only slave
Date
Msg-id CAG1_KcCmUguyZA0WiQmCbVg6YeVB=OLY7VgJJPbZwYh2Ao+Gzw@mail.gmail.com
Whole thread Raw
In response to Re: Merge a sharded master into a single read-only slave  (Sébastien Lorion <sl@thestrangefactory.com>)
List pgsql-general


On Thu, Jun 5, 2014 at 2:09 PM, Sébastien Lorion <sl@thestrangefactory.com> wrote:
On Thu, Jun 5, 2014 at 12:55 PM, Francisco Olarte <folarte@peoplecall.com> wrote:
Hi Sébastien:

On Thu, Jun 5, 2014 at 5:41 PM, Sébastien Lorion
<sl@thestrangefactory.com> wrote:

> .... Correct me if I am wrong, but will it not also suffer the same
> limitation as any statement based replication, namely that the "merged"
> slave will have to sustain the same write load as all shards combined ?

I cannot tell you the exact mimeo behaviour, but if you incremental
replication using an id/timestamp  by >pulling< changes from the
masters, you will normally batch them and insert all the changes to
the slaves in a single transaction, which leads to less load as many
times your limit is in transaction rate, not record rate. (i.e., every
5 minutes you query for all the tuples changed, and insert/update them
all in one go ) ( Also, if tuples are updated many times between
sweeps the slave will get only one )

Francisco Olarte.

​You are right, requesting changes at fixed time intervals would certainly help reduce the load. I will have to test and see if a good balance can be achieved between not having stale data for too long and keeping up with writes.

Sébastien


If you have any questions while evaluating it, feel free to ask or post any issues to github.

--
Keith Fiske
Database Administrator
OmniTI Computer Consulting, Inc.
http://www.keithf4.com

pgsql-general by date:

Previous
From: David G Johnston
Date:
Subject: Re: Optimizer issue -- bad query plan?
Next
From: Moshe Jacobson
Date:
Subject: Re: Optimizer issue -- bad query plan?