Re: PITR logging control program - Mailing list pgsql-hackers

On Thu, 2004-04-29 at 20:24, Bruce Momjian wrote:
> Simon Riggs wrote:
> > 
> > The archiver should be able to do a whole range of things. Basically,
> > that point was discussed and the agreed approach was to provide an API
> > that would allow anybody and everybody to write whatever they wanted.
> > The design included pg_arch since it was clear that there would be a
> > requirement in the basic product to have those facilities - and in any
> > case any practically focused API has a reference port as a way of
> > showing how to use it and exposing any bugs in the server side
> > implementation.
> > 
> > The point is...everybody is now empowered to write tape drive code,
> > whatever you fancy.... go do.
> 
> Agreed we want to allow the superuser control over writing of the
> archive logs.  The question is how do they get access to that.  Is it by
> running a client program continuously or calling an interface script
> from the backend?
> 
> My point was that having the backend call the program has improved
> reliablity and control over when to write, and easier administration.
> 

Agreed. We've both suggested ways that can occur, though I suggest this
is much less of a priority, for now. Not "no", just not "now".

> How are people going to run pg_arch?  Via nohup?  In virtual screens? If
> I am at the console and I want to start it, do I use "&"?  If I want to
> stop it, do I do a 'ps' and issue a 'kill'?  This doesn't seem like a
> good user interface to me.
> 
> To me the problem isn't pg_arch itself but the idea that a client
> program is going to be independently finding(polling) and copying of the
> archive logs.
> 
> I am thinking the client program is called with two arguments, the xlog
> file name, and the arch location defined in GUC.  Then the client
> program does the write.  The problem there though is who gets the write
> error since the backend will not wait around for completion?
> 
> Another case is server start/stop.  You want to start/stop the archive
> logger to match the database server, particularly if you reboot the
> server.  I know Informix used a client program for logging, and it was a
> pain to administer.
> 

pg_arch is just icing on top of the API. The API is the real deal here.
I'm not bothered if pg_arch is not accepted, as long as we can adopt the
API. As noted previously, my original mind was to split the API away
from the pg_arch application to make it clearer what was what. Once that
has been done, I encourage others to improve pg_arch - but also to use
the API to interface with other BAR prodiucts.

If you're using PostgreSQL for serious business then you will be using a
serious BAR product as well. There are many FOSS alternatives...

The API's purpose is to allow larger, pre-existing BAR products to know
when and how to retrieve data from PostgreSQL. Those products don't and
won't run underneath postmaster, so although I agree with Peter's
original train of thought, I also agree with Tom's suggestion that we
need an API more than we need an archiver process. 

I would be happy with an exteral program if it was started/stoped by the
> postmaster (or via GUC change) and received a signal when a WAL file was
> written.  

That is exactly what has been written.

The PostgreSQL side of the API is written directly into the backend, in
xlog.c and is therefore activated by postmaster controlled code. That
then sends "a signal" to the process that will do the archiving - the
Archiver side of the XLogArchive API has it as an in-process library.
(The "signal" is, in fact, a zero-length file written to disk because
there are many reasons why an external archiver may not be ready to
archive or even up and running to receive a signal).

The only difference is that there is some confusion as to the role and
importance of pg_arch.

Best Regards, Simon Riggs