Re: The plan for FDW-based sharding - Mailing list pgsql-hackers

From Oleg Bartunov
Subject Re: The plan for FDW-based sharding
Date
Msg-id CAF4Au4x4VNVZTArT2A3Gtyg=c7s=5qwdtie2X4gwJbR8jAZDLw@mail.gmail.com
Whole thread Raw
In response to Re: The plan for FDW-based sharding  (Robert Haas <robertmhaas@gmail.com>)
List pgsql-hackers


On Tue, Mar 1, 2016 at 7:03 PM, Robert Haas <robertmhaas@gmail.com> wrote:
On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Tue, Mar  1, 2016 at 10:19:45AM -0500, Robert Haas wrote:
>> > Two reasons:
>> > 1. There is no ideal implementation of DTM which will fit all possible needs
>> > and be  efficient for all clusters.
>>
>> Hmm, what is the reasoning behind that statement?  I mean, it is
>> certainly true that there are some places where we have decided that
>> one-size-fits-all is not the right approach.  Indexing, for example.
>
> Uh, is that even true of indexing?  While the plug-in nature of indexing
> allows for easier development and testing, does anyone create plug-in
> indexing that isn't shipped by us?  I thought WAL support was something
> that prevented external indexing solutions from working.

True.  There is an API, though, and having pluggable WAL support seems
desirable too.  At the same time, I don't think we know of anyone
maintaining a non-core index AM ... and there are probably good
reasons for that.  We end up revising the index AM API pretty

We'd love to develop new special index AM, that's why we all are for pluggable WAL. I think there are will be other AM developers, once we open the door for that.
 
regularly every time somebody wants to do something new, so it's not
really a stable API that extensions can just tap into.  I suspect that
a transaction manager API would end up similarly situated.

I don't expect many other TM developers, so there is no problem with improving API. We started from practical needs and analyses of many academical papers. We spent a year to play with several prototypes to prove our proposed API (expect more in several months). Everybody could download them a test. Wish we can do that with FDW-based sharding solution.

Of course, we can fork postgres as XC/XL people did and certainly eventually will do, if community don't accept our proposal, since it's very difficult to work on cross-releases projects. But then there are will be no winners, so why do we all are aggressively don't understand each other ! I was watching  XC/XL for years and thought I don't want to go this way of isolation from the community, so we decided to let TM pluggable to stay with community and let everybody prove their concepts. if you have ideas how to improve TM API, we are open, if you know it's broken by design, let's help us to fix it.  I have my understanding about FDW, but I deliberately don't participate in some very hot discussion, just because I feel myself not commited to work on. Your group is very enthusiastic on FDW, it's ok until you improve FDW in general way, I'm very happy on current work.  I prefer you show prototype of sharding solution, which convince us in functionality and perfromance. I agree with Thomas Vondra, that we don't want to wait for years to see the result, we want to expect results, based on prototype, which should be done between releases. If you don't have enough resources for this, let's do together with community.  Nobody as I've seen are against FDW sharding, people complained about "the only sharding solution" in postgres, without proving so.
 


 

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

pgsql-hackers by date:

Previous
From: Ashutosh Bapat
Date:
Subject: Re: Issue with NULLS LAST, with postgres_fdw sort pushdown
Next
From: Oleg Bartunov
Date:
Subject: Re: The plan for FDW-based sharding