Thread: Plan B for log rotation support: borrow Apache code

Plan B for log rotation support: borrow Apache code

From
Tom Lane
Date:
Something we've discussed repeatedly is adding a log rotation program to
the distribution so that we don't have to hand-wave in the part of the
documentation that talks about setting up non-syslog-based logging.
A few weeks ago Andrew Sullivan offered the rotator his company uses,
but there were various objections to it (mainly that it couldn't coexist
with pg_ctl, IIRC).

I think a reasonable Plan B is to borrow Apache's rotatelogs program
hook, line, and sinker, and include it as a contrib module.  It's small
and it does exactly what we want: read stdin and drop it into a series
of text files.  The license can be read at
http://www.apache.org/LICENSE.txt
It's essentially BSD terms; I think all we would need to do differently
from what we'd do with homegrown code is include the text of the Apache
license in the README.rotatelogs documentation file that the contrib
module would install.

It would take an hour or two's work to put together a standalone version
of rotatelogs.c (one that doesn't depend on the Apache Portable Runtime
library).  Seems well worth it to me.  Does anyone have an objection to
doing this?
        regards, tom lane


Re: Plan B for log rotation support: borrow Apache code

From
Peter Eisentraut
Date:
Tom Lane writes:

> It would take an hour or two's work to put together a standalone version
> of rotatelogs.c (one that doesn't depend on the Apache Portable Runtime
> library).  Seems well worth it to me.  Does anyone have an objection to
> doing this?

Or we could take the one I posted, which does exactly the same thing as
Apache's (at least about a year ago, not sure where the APR plays in
nowadays), thus shortcircuiting that porting effort.

-- 
Peter Eisentraut   peter_e@gmx.net



Re: Plan B for log rotation support: borrow Apache code

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> Or we could take the one I posted, which does exactly the same thing as
> Apache's (at least about a year ago, not sure where the APR plays in
> nowadays), thus shortcircuiting that porting effort.

I'd forgotten that one.  Where is it again?
        regards, tom lane


Re: Plan B for log rotation support: borrow Apache code

From
Bruce Momjian
Date:
Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Or we could take the one I posted, which does exactly the same thing as
> > Apache's (at least about a year ago, not sure where the APR plays in
> > nowadays), thus shortcircuiting that porting effort.
> 
> I'd forgotten that one.  Where is it again?

I just bounced it the hackers list.  It has an April date.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: Plan B for log rotation support: borrow Apache code

From
Bruce Momjian
Date:
Here are the log scripts posted by Andrew Sullivan and Peter E.

---------------------------------------------------------------------------

Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Or we could take the one I posted, which does exactly the same thing as
> > Apache's (at least about a year ago, not sure where the APR plays in
> > nowadays), thus shortcircuiting that porting effort.
>
> I'd forgotten that one.  Where is it again?
>
>             regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
#!/usr/local/bin/perl
#
#pglog-rotator
#Copyright (c) Liberty RMS 2001-2003
#This file may be distributed under the same terms as PostgreSQL
#This file comes with NO WARRANTY WHATSOEVER
#
#

use IPC::Open3;
use IO::Handle;

$err_log_file='/path/to/logs/stderr_pg.log';
$out_log_file='/path/to/logs/stdout_pg.log';

#### log size in MBytes
$size_limit=1;

#### rotate nn files
$max_logs=7;

#### postgres cmd
# it is best if you set $PGDATA
# and use the $PATH to call just postmaster.
# That makes the output of 'ps -auxww' more valuable
$command = 'postmaster';


#######
sub HUP_SIGNAL {
    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(time);
    $gmtime = sprintf ("%.2d/%.2d/%.4d %.2d:%.2d:%.2d: ",  $mon+1, $mday, 1900+$year, $hour, $min, $sec);
        print "$gmtime: SIGHUP received. Ignored.\n";
}
$SIG{HUP} = \&HUP_SIGNAL;


open(LOG, ">>$err_log_file");
autoflush LOG 1;
open(OUT_LOG, ">>$out_log_file");
autoflush OUT_LOG 1;

$cnt = $max_logs;
$size_limit=$size_limit * 1024 * 1024;

### postgres 7.1.3 writes to stderr in any debug_level (0 to 16)
sub read_stderr
{
    while(<ERRFH>)
    {
    ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat($err_log_file);
    if ( $size > $size_limit ) {
        close LOG;

        while ($cnt > 0) {
            $old_log=sprintf("%s.%s", $err_log_file, $cnt);
            $new_log=sprintf("%s.%s", $err_log_file, $cnt-1);
            rename $new_log, $old_log;
            $cnt--;
        }
        rename $err_log_file, $new_log;
        $cnt = $max_logs;
        open(LOG, ">>$err_log_file");
    }
    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(time);
    $gmtime = sprintf ("%.2d/%.2d/%.4d %.2d:%.2d:%.2d: ",  $mon+1, $mday, 1900+$year, $hour, $min, $sec);
    print LOG "$gmtime $_";
    }

}

### postgres 7.1.3 writes to stdout in debug_level 3 to 16
sub read_stdout
{
    while(<RDRFH>)
    {
    ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat($out_log_file);
    if ( $size > $size_limit ) {
        close OUT_LOG;

        while ($cnt > 0) {
            $old_log=sprintf("%s.%s", $out_log_file, $cnt);
            $new_log=sprintf("%s.%s", $out_log_file, $cnt-1);
            rename $new_log, $old_log;
            $cnt--;
        }
        rename $out_log_file, $new_log;
        $cnt = $max_logs;
        open(OUT_LOG, ">>$out_log_file");
    }
    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(time);
    $gmtime = sprintf ("%.2d/%.2d/%.4d %.2d:%.2d:%.2d: ",  $mon+1, $mday, 1900+$year, $hour, $min, $sec);
    print OUT_LOG "$gmtime $_";
    }

}


$pid = open3(\*WTRFH, \*RDRFH, \*ERRFH, $command);

printf "postmaster [$pid] started: ";

print "$command\n";
print "STDERR log file: $err_log_file\n";
print "STDOUT log file: $out_log_file\n";
printf ("size limit: %.3f MBytes, ", $size_limit / 1024 / 1024);
print "max logs: $max_logs\n";


# read stdout from postmaster
if (fork() == 0) {    ## this is the child
    read_stdout();
    exit(0);
}

read_stderr();

printf "postmaster [$pid] killed.\n\n";

#include "c.h"

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
#include <errno.h>
#include <unistd.h>

#include "pqsignal.h"

volatile static int hangup_flag = 0;


static void
signalhandler(SIGNAL_ARGS)
{
    hangup_flag = 1;
}


#define BUF_SIZE 8192
#define MAX_ERRORS 200

#define MAX_ERRORS_CHECK() do { errcount++; if (max_errors > 0 && errcount >= max_errors) exit(2); } while(0)

int
main(int argc, char *argv[])
{
    const char *filename;
    int fd = -1;
    static char buf[BUF_SIZE];
    unsigned int errcount = 0;
    unsigned int max_errors = MAX_ERRORS;

    if (argc != 2)
    {
        fprintf(stderr, "%s: missing required argument\n", argv[0]);
        fprintf(stderr, "Try '%s --help' for more information.\n", argv[0]);
        exit(1);
    }

    if (strcmp(argv[1], "--help")==0)
    {
        printf("this should be a help message...\n");
        exit(0);
    }

    filename = argv[1];

    fd = open(filename, O_WRONLY | O_CREAT | O_NOCTTY | O_APPEND, 0666);
    if (fd < 0)
    {
        fprintf(stderr, "%s: could not open file %s: %s\n",
                argv[0], filename, strerror(errno));
        exit(1);
    }

    pqsignal(SIGUSR1, signalhandler);

    for (;;)
    {
        int read_bytes;
        int written_bytes;
        char * buf_ptr;

        read_bytes = read(0, buf, sizeof(buf));
        if (read_bytes < 0)
        {
            if (errno == EINTR)
                continue;
            else
            {
                fprintf(stderr, "*** %s: read error from %s: %s\n",
                        argv[0], filename, strerror(errno));
                MAX_ERRORS_CHECK();
            }
        }
        if (read_bytes == 0)
        {
            /* end of file, postmaster exited? */
            close(fd);
            exit(0);
        }

        if (hangup_flag)
        {
            int fdnew;

            fdnew = open(filename, O_WRONLY | O_CREAT | O_NOCTTY | O_APPEND, 0666);
            if (fdnew < 0)
            {
                fprintf(stderr, "*** %s: could not open new output file %s: %s\n",
                       argv[0], filename, strerror(errno));
                MAX_ERRORS_CHECK();
            }
            else
            {
                close(fd);
                fd = fdnew;
            }
            hangup_flag = 0;
        }

        buf_ptr = buf;
        do
        {
            written_bytes = write(fd, buf_ptr, read_bytes);
            if (written_bytes < 0)
            {
                if (errno == EINTR)
                    continue;
                fprintf(stderr, "*** %s: could not write to file %s: %s\n",
                        argv[0], filename, strerror(errno));
                MAX_ERRORS_CHECK();

                break;
            }
            if (written_bytes < read_bytes)
            {
                buf_ptr += written_bytes;
                read_bytes -= written_bytes;
                continue;
            }
            break;
        } while(1);
    }

    return 127;
}

Re: Plan B for log rotation support: borrow Apache code

From
"Andrew Dunstan"
Date:
Plan B.0.1 might be to get the version of rotatelogs.c from the Apache 1.3
branch so that removing the APR stuff wasn't necessary (IIRC APR is a purely
2.x thing in Apache).

Can be viewed here:
http://cvs.apache.org/viewcvs.cgi/apache-1.3/src/support/rotatelogs.c

Or else use the one from Peter, I don't care, just pointing out that
removing APR shouldn't be necessary.

andrew (big fan of not reinventing the wheel - good programmers write code,
great programmers steal code ;-)

----- Original Message ----- 
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: <pgsql-hackers@postgresql.org>
Sent: Friday, May 23, 2003 3:13 PM
Subject: [HACKERS] Plan B for log rotation support: borrow Apache code


> Something we've discussed repeatedly is adding a log rotation program to
> the distribution so that we don't have to hand-wave in the part of the
> documentation that talks about setting up non-syslog-based logging.
> A few weeks ago Andrew Sullivan offered the rotator his company uses,
> but there were various objections to it (mainly that it couldn't coexist
> with pg_ctl, IIRC).
>
> I think a reasonable Plan B is to borrow Apache's rotatelogs program
> hook, line, and sinker, and include it as a contrib module.  It's small
> and it does exactly what we want: read stdin and drop it into a series
> of text files.  The license can be read at
> http://www.apache.org/LICENSE.txt
> It's essentially BSD terms; I think all we would need to do differently
> from what we'd do with homegrown code is include the text of the Apache
> license in the README.rotatelogs documentation file that the contrib
> module would install.
>
> It would take an hour or two's work to put together a standalone version
> of rotatelogs.c (one that doesn't depend on the Apache Portable Runtime
> library).  Seems well worth it to me.  Does anyone have an objection to
> doing this?
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org