Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader - Mailing list pgsql-hackers

From Andres Freund
Subject Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader
Date
Msg-id 201209171406.26781.andres@2ndquadrant.com
Whole thread Raw
In response to Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader  (Heikki Linnakangas <hlinnakangas@vmware.com>)
List pgsql-hackers
On Monday, September 17, 2012 01:50:33 PM Heikki Linnakangas wrote:
> On 17.09.2012 14:42, Andres Freund wrote:
> > On Monday, September 17, 2012 12:55:47 PM Heikki Linnakangas wrote:
> >> On 17.09.2012 13:01, Andres Freund wrote:
> >>> On Monday, September 17, 2012 11:07:28 AM Andres Freund wrote:
> >>>> On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote:
> >>>>> On 17.09.2012 11:12, Andres Freund wrote:
> >>>>>> On Monday, September 17, 2012 09:40:17 AM Heikki Linnakangas wrote:
> >>>>>> If you don't want the capability to forward/filter the data and read
> >>>>>> partial data without regard for record constraints/buffering your
> >>>>>> patch seems to be quite a good start. It misses xlogreader.h
> >>>>>> though...
> >>>>> 
> >>>>> Ah sorry, patch with xlogreader.h attached.
> >>>> 
> >>>> Will look at it in a second.
> >>> 
> >>> It seems we would need one additional callback for both approaches
> >>> like:
> >>> 
> >>> ->error(severity, format, ...)
> >>> 
> >>> For both to avoid having to draw in elog.c.
> >> 
> >> Yeah. Another approach would be to return the error string from
> >> ReadRecord. The caller could then do whatever it pleases with it, like
> >> ereport() it to the log or PANIC. I think I'd like that better.
> > 
> > That seems a bit more complex from a memory management perspective as you
> > probably would have to sprintf() into some buffer. We cannot rely on a
> > backend environment with memory contexts around et al...
> 
> Hmm. I was thinking that making this work in a non-backend context would
> be too hard, so I didn't give that much thought, but I guess there isn't
> many dependencies to backend functions after all. palloc/pfree are
> straightforward to replace with malloc/free. 
Hm. I thought that it was pretty much a design requirement that this is usable 
outside of the backend environment?

> That's what we could easily do with the error messages too, just malloc a
> suitably sized buffer.
Not very comfortable though... Especially if you need to return an error from 
the read_page callback...


> How does a non-backend program get access to xlogreader.c? Copy
> xlogreader.c from the source tree at build time and link into the
> program? Or should we turn it into a shared library?
Not really sure. I thought about just putting it in pgport or such, but that 
seemed ugly as well.
The bin/xlogdump hack, which I find really helpful, at first simply had a 
dependency on ../../backend/access/transam/xlogreader.o which worked fine. Till 
it needed more because of *_desc routines... But Alvaro started to work on this 
although I don't know when he will be able to finish it.


Greetings,

Andres
-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services



pgsql-hackers by date:

Previous
From: Heikki Linnakangas
Date:
Subject: Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader
Next
From: Dave Page
Date:
Subject: Re: Possible fix for occasional failures on castoroides etc