Re: Re: xReader, double-effort (was: Temporary tables under hot standby) - Mailing list pgsql-hackers

Andres Freund <andres@2ndquadrant.com> wrote:
> In the current, prototypal, state there is one component thats
> integrated into the server (because it needs information thats
> only available there).
The xReader design was based on the idea that it would be nice not
to cause load on the master machine, and that by proxying the WAL
stream to the HS, using synchronous replication style to write from
xReader to the HS, you could use the HS for a source for that data
with it being at exactly the right point in time to query it.
I'm not convinced that I would rather see the logic fixed inside the
master as opposed to being deployable on the master's machine, the
slave machine, or even on its own machine in between.
> That component is layered ontop of a totally generic xlog
> reading/parsing library that doesn't care at all where its
> running.
That's cool.
> Its also used in another cluster to read the received (filtered)
> stream.
I don't quite follow what you're saying there.
> I plan to submit the XLogReader (thats what its called atm)
> before everything else, so everybody can take a look as soon as
> possible.
Great!  That will allow more discussion and planning.
> I took a *very* short glance over the current wiki description of
> xReader and from that it seems to me it would benefit from trying
> to make it architecturally more similar to the rest of pg.
We're planning on using existing protocol to talk between pieces. 
Other than breaking it out so that it can run somewhere other than
inside the server, and allowing clients to connect to xReader to
listen to WAL events of interest, are you referring to anything
else?
> I also would suggest reviewing how the current walreceiver/sender,
> and their protocol, work.
Of course!  The first "inch-stone" in the GSoC project plan
basically consists of creating an executable that functions as a
walreceiver and a walsender to just pass things through from the
master to the slave.  We build from there by allowing clients to
connect (again, over existing protocol) and register for events of
interest, and then recognizing different WAL records to generate
events.  The project was just going to create a simple client to
dump the information to disk, but with the time saved by adopting
what you've already done, that might leave more time for generating
a useful client.
Aakash, when you get a chance, could you fill in the "inch-stones"
from the GSoC proposal page onto the Wiki page?  I think the
descriptions of those interim steps would help people understand
your proposal better.  Obviously, some of the particulars of tasks
and the dates may need adjustment based on the new work which is
expected to appear before you start, but what's there now would be a
helpful reference.
-Kevin