On Tue, Aug 16, 2011 at 00:05, Simon Riggs <simon@2ndquadrant.com> wrote:
> On Mon, Aug 15, 2011 at 10:32 PM, Magnus Hagander <magnus@hagander.net> wrote:
>
>>> At present the WALSender only sends from one file at a time, so
>>> sending a message when we open a new file would be straightforward.
>>
>> Are you sure? We can receive a single message spanning multiple files...
>
> You're right. That was the way the original code ran but I thought we
> had stopped that when we introduced MAX_SEND_SIZE. The comment says
> "don't cross a logfile boundary within one message". What that
> actually does is prevent us incrementing a logid value, which happens
> every 255 files. I read that as meaning "WAL file" which is not what
> it means at all.
That has bitten me many times. log file != wal file.. It's not exactly
intuitive.
> So right now what we do is allow a single packet to span multiple
> files, but since MAX_SEND_SIZE is 128KB it will always be smaller than
> a single file, so we can only ever span two files at most.
Unless someone has tweaked their xlog segment size to insane values.
I ended up writing a loop that can deal with it spanning >2 files as
well - but that also turned out to be less code than the special case
for 2 files, so I'll be keeping that :-)
> That is all just a little bizarre, especially since libpq sends data
> in 8KB chunks anyway.
>
> So I suggest we change XLogSend() so that it only ever sends up to the
> end of a file. That way all "w" messages will relate to just one file,
> and we can have another message to initiate a new file. And then, as
> you say, give full metadata for the new file.
That could be an idea in itself. It is not necessary for the log
receiver, since it can use the #define FRONTEND hack (which only works
if you build inside the source tree of course, not if it's an
extension) but it might make the protocol simpler to deal with for
third parties who want to do something similar.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
