Re: [PATCHES] serverlog rotation/functions - Mailing list pgsql-hackers

Andreas Pflug wrote:
> Bruce Momjian wrote:
> > OK, I talked to Tom about this patch and I understand the issues now.
> > 
> > I think the best solution will be to have the postmaster start a child
> > process that can read the guc variables and create a log file based on
> > it contents.  The child would be responsible to create a new log file
> > every X seconds as specified in the postgresql.conf file.
> > 
> 
> OK, next round.
> 
> > Backends wishing to read the log file would call a function to list the
> > contents of a directory.  We can do this by creating a generic backend
> > super-user-only function that can list the contents of any directory. 
> 
> If logfiles are located using directory listings, this would mean we 
> should have well-known filename patterns instead of the current template 
> stuff. So only the serverlog directory name would be selectable, if that 
>   makes still sense (I'd opt to drop that).

I don't see any reason to have a pattern though I suppose if you mix
pgsql log files in with other log files it might be a problem.  One idea
would be for the client-side program to do some processing like this:
SELECT *FROM dir_listing('/var/log') AS dirWHERE dir LIKE 'pgsql_%'

or something like that where the client pulls apart the directory
specifiction like this:
log_dest = '/var/log/postgresql.log.%Y-%m-%d_%H%M%S'

You do something that splits the value into directory name and file name
and removes every letter after %.
/var/logpostgresql.log.%-%-%_%%%

Another idea is to allow filename wildcards in the listing so it
becomes:
SELECT *FROM dir_listing('/var/log/postgresql.log.*-*-*_***') AS dir

While that is nice, it doesn't match the functionality of opendir so we
are perhaps better with one that doesn't handle wildcards and we just do
the wildcard processing in the WHERE clause.

> > Then you would use an API to read a specific file, similar to the log
> > reading functions you already have.  In fact, you could set up the API
> > so reading a directory would return a list of directory names so you
> > don't need a separate directory listing function.  (Does your API return
> > file contents as one big text string or as lines.  If you return it as
> > one big text string, the directory idea will not work and you will need
> > a separate function.)
> 
> We'd better have distinct functions. The current is for reading large 
> chunks of data without interpretation of contents, and dir listing might 
>   need tweaking for win32.

OK.

> Now I wonder if it's better to have generic superuser-only functions for 
> list/read/write/delete (this would enable client-side editing of 
> postgresql.conf), or tie the functions to logfiles only (in which case 
> the logfile functions shouldn't use filenames but logfile timestamps).

Agreed, that was one of my goals.  Right now the superuser can read any
file with COPY (read it as text lines into a text table).  Allowing
directory listings is a natural extension.

> I still believe the log process should write the current logfile 
> filename to shared-mem (write-only), so backends can prevent deleting it.

Once we do this there will not be any backend writing to those files.
(We will need the log subprocess pid in shared memory so backends can
send signals to it.)  I am not sure how we will do file deletes but I
think each backend will have to do the delete itself rather than try to
pass the file name to the log process.  I think we will have to assume
the log file names increase in ordering so we know which one is the
current one.  I can't think if a cleaner way to communicate this to the
backends except perhaps as you suggest as shared memory areas that
contains the name, but we will need a lock so the backends don't read it
while it is changing.  That would be a nice feature.

> > So, we have:
> > 
> >     o   use pipe and dup to copy stdout and stderr to your
> >         postmaster child
> >     o   new guc variables to specify log destination and rotation times
> >     o   a server-side function to force a log rotate
> >     o   API to list a directory and show file contents
> 
> I'll start on the log process.

Sorry to be dumping more work on you but I think this is a better
directory to go for this feature.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073