Re: Merge a sharded master into a single read-only slave - Mailing list pgsql-general

From Sébastien Lorion
Subject Re: Merge a sharded master into a single read-only slave
Date
Msg-id CAGa5y0OnCW5-XO67Y+07iBccE71658n6dJ032svvytxkBEd-8A@mail.gmail.com
Whole thread Raw
In response to Re: Merge a sharded master into a single read-only slave  (Keith Fiske <keith@omniti.com>)
Responses Re: Merge a sharded master into a single read-only slave
List pgsql-general
On Wed, Jun 4, 2014 at 1:50 PM, Keith Fiske <keith@omniti.com> wrote:
Not sure if this will work for you, but sharing a similar scenario in case it may work for you.

An extension I wrote provides similar logical replication as you've probably seen in other tools.
https://github.com/omniti-labs/mimeo
One difference is it has several methods for this replication, one being incremental based on either time or serial ID. Since incremental replication requires just read-only access on the source databases, it causes no extra write overhead as most logical replication solutions do (triggers writing to queue tables).

A client of ours had a table sharded by UUID to 512 clusters but needed that data pulled to a single cluster for reporting purposes. The tables also had a timestamp column that was set on each insert/update, so the incremental replication method was able to be used here to pull data from all clusters to a single cluster. The single reporting cluster then just had an inheritance table set up with an empty parent table pointing to all the child tables that pulled data into them.

Yes, it was a lot of setup since each of the 512 tables has to be set up individually. But once it was set up it worked surprisingly well. And it's honestly a use case I had never foreseen for the extension.

Not sure if this would work in your case, but maybe it can at least give you an idea of what can be done.

--
Keith Fiske
Database Administrator
OmniTI Computer Consulting, Inc.
http://www.keithf4.com


​Thank you Keith for taking the time to let me know about your solution. It looks great indeed, especially the part about not putting load on the shards themselves. Correct me if I am wrong, but will it not also suffer the same limitation as any statement based replication, namely that the "merged" slave will have to sustain the same write load as all shards combined ?

Sébastien

pgsql-general by date:

Previous
From: Carlos Carcamo
Date:
Subject: Re: help with a procedure
Next
From: Seref Arikan
Date:
Subject: Re: help with a procedure