Re: Horizontal scalability/sharding - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: Horizontal scalability/sharding
Date
Msg-id CAA4eK1KuA8d=S1fKiMFRfw1PmX9e-2E1y4pN_AH6B8bf_485GQ@mail.gmail.com
Whole thread Raw
In response to Re: Horizontal scalability/sharding  (Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>)
Responses Re: Horizontal scalability/sharding  (Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>)
List pgsql-hackers
On Wed, Sep 2, 2015 at 11:35 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
On 2015/09/02 14:28, Amit Langote wrote:
On 2015-09-02 PM 01:28, Amit Kapila wrote:
On Tue, Sep 1, 2015 at 9:48 PM, Robert Haas <robertmhaas@gmail.com> wrote:
I'm not averse to making the "connect to the remote nodes" part of
this solution use something other than the FDW infrastructure at some
point in time if somebody's prepared to build something better.  On
the other hand, I think it's extremely clear that the FDW
infrastructure has a large amount of potential upon which we have
thoroughly failed to capitalize.  Patches have already been written
for UPDATE/DELETE pushdown and for join pushdown.

Will pushing down writes (Update/Delete) sufficient to maintain sane locking
behaviour and deadlock detection that can occur during writes on multiple
shards?  For example it could easily be the case where a single Update
statement could effect multiple shards and cause deadlock due to waits
across the nodes.  Now unless we have some distributed lock manager or
some other way to know the information of locks that happens across
shards, it could be difficult to detect deadlocks.

I wonder if Ashutosh's atomic foreign transactions patch would address any
issues inherent in such cases...

The UPDATE/DELETE pushdown, which I've proposed, would ensure the sane behaviour for inherited UPDATEs/DELETEs, as existing non-pushed-down UPDATE/DELETE does, because inheritance_planner guarantees that all backends lock inheritance children in the same order to avoid needless deadlocks.


Will it be able to do it for row level locks, row level locking occurs
during updation of a row, so will it be possible to ensure the order of
locks on rows?

Will it handle deadlocks across different table partitions. Consider
a case as below:

T1
1. Updates row R1 of T1 on shard S1
2. Updates row R2 of T2 on shard S2

T2
1. Updates row R2 of T2 on shard S2 
2. Updates row R1 of T1 on shard S1



With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

pgsql-hackers by date:

Previous
From: Michael Paquier
Date:
Subject: Re: Fwd: Core dump with nested CREATE TEMP TABLE
Next
From: Haribabu Kommi
Date:
Subject: Re: Multi-tenancy with RLS