Re: Horizontal scalability/sharding - Mailing list pgsql-hackers

From Tomas Vondra
Subject Re: Horizontal scalability/sharding
Date
Msg-id 55E84DD8.6000703@2ndquadrant.com
Whole thread Raw
In response to Re: Horizontal scalability/sharding  (Amit Kapila <amit.kapila16@gmail.com>)
Responses Re: Horizontal scalability/sharding
List pgsql-hackers
Hi,

On 09/03/2015 05:02 AM, Amit Kapila wrote:
> On Thu, Sep 3, 2015 at 8:28 AM, Bruce Momjian <bruce@momjian.us
> <mailto:bruce@momjian.us>> wrote:
>  >
>  > On Wed, Sep  2, 2015 at 07:50:25PM -0700, Joshua Drake wrote:
>  > > >Can you explain why logical replication is better than binary
>  > > >replication for this use-case?
>  > > >
>  > >
>  > > Selectivity?
>  >
>> I was assuming you would just create identical slaves to handle
>> failure, rather than moving selected data around.
>  >
>
> Yes, I also think so, otherwise when the shard goes down and it's
> replica has to take the place of shard, it will take more time to
> make replica available as it won't have all the data as original
> shard had.

Not really, the idea is that you don't need to create the replica 
immediately. The system recognizes that primary shard location is 
unavailable and redirects the tasks to the "replicas." So the time to 
recreate the failed node is not that critical.

It needs to be done in a smart way to prevent some typical issues like 
suddenly doubling the load on replicas due to failure of the primary 
location. By using different group of nodes for each "data segment" you 
can eliminate this, because the group of nodes to handle the additional 
load will be larger.

The other issue then of course is that the groups of nodes must not be 
entirely random, otherwise the cluster would suffer data loss in case of 
outage of arbitrary group of K nodes (where K is the number of replicas 
for each piece of data).

It's also non-trivial to do this when you have to consider racks, data 
centers etc.

With regular slaves you can't do any of this - no matter what you do, 
you have to load balance the additional load only on the slaves.

regards

--
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



pgsql-hackers by date:

Previous
From: Anastasia Lubennikova
Date:
Subject: Re: [PATCH] Microvacuum for gist.
Next
From: Michael Paquier
Date:
Subject: Re: Fwd: Core dump with nested CREATE TEMP TABLE