Re: Archival API - Mailing list pgsql-hackers-pitr

>>Raymond Siebert wrote
>>Simon Riggs wrote:
>>So in summary, the API is:
>>Archiver initialises and waits on notify
>>Postgresql initialises
>>...then
>>Postgresql fills log, switches and close it, then calls
>>XLogArchiveNotify()
>>Archiver moves log somewhere safe, then sets state such that...
>>...sometime later
>>Postgresql checks XLogArchiveBusy() to see if its safe to recycle file
>>and discovers the state set by
>>
>>API is completely unintrusive on current tried and tested operation,
and
>>leaves the archiver(s) free to act as they choose, outside of the
>>address space of PostgreSQL. That way we don't have to update
regession
>>tests with some destructive non-manual crash tests to show that works.
>>
>>Clearly, we wouldn't want WAL logs to hang around too long, so we need
>>an initiation method for the archival process. Otherwise, we'll be
>>writing "nnn.full" notifications yet without anybody ever deleting
them.
>>Either this could be set at startup with an archive_log_mode parameter
>>(OK, the names been used before, but if the cap fits, wear it) or
>>setting a maximum limit to number of archive logs and a few other
ideas,
>>none of which I like.
>
>Another idea...

Thank you very much for your input, it is gratefully received.

>A fixed classification of events and a defined framework in user
supplied >program is the way to success.

Yes, I agree. I'm working on a better description of the overall design,
to allow everybody to think it through with me. Should be next week
now...

>Another idea is an event handling system in database system.
>Sending defined signals to the 'outer world' through an API to a user
>supplied program.
>This user supplied program can later on handle other events in addition
to >backup of WAL logs.

That was an approach I considered. One aspect of PITR is that testing is
a very important part of the overall solution. With that in mind, a very
tightly defined, very specific API seems the best way to go. A more
general facility might get mixed up and end up breaking the PITR logic.

External notification is possible (I think, never tried) by using user
defined C language functions. It should be fairly straightforward to use
those to get/put on message queues, send POSIX signals etc.

>The archiver process must have a naming convention for archived WAL
logs - >especially for recovery.

Yes. I would suggest the database name as a prefix to the WAL log file
name. The log file name is a very long string of digits that "never
repeats" according to the manual. I'll cross that bridge when it
happens. That should guarantee uniqueness **within** an organisation,
(assuming there is a unique database naming policy is place) which is
hopefully the limit of archived log files anyway.

>Archiving WAL logs may require an archive destination - also needed for
>recovery.

That would be the user-supplied program's role to define that. The
implementation will clearly require a reference implementation of just
such a program, to allow testing.

>Essential input to user supplied program is the full-pathed archived
WAL >log.
>A user supplied backup program backups full-pathed archived WAL log to
an >whatever Backup Destination (even /dev/null for suicide characters).

>After successfull backup the archived WAL log needs to be removed.
>All is handled internally in the user supplied program.

That's what the basic design proposed, yes.

>Advantage of this idea is that backup of archived WAL logs is
automatically >enabled without the manual work of an DB Admin person.

I think it is important that archival can only be activated by explicit
command from the database administrator. If not, then it could be some
kind of security hole to start sniffing a log without permission. Your
input is good though, because I haven't given security much thought yet
and you have spurred me to consider this further.

I hope I can call on you soon to review the design docs and later to
assist with testing?

More input welcome!

Best Regards, Simon Riggs