Thread: more contrib: log rotator
Since now is the time for contrib/ flamewars, this seemed a good time to suggest this. My colleague, Sorin Iszlai, wrote us a little program for rotating our Postgres logs. It reads stdout and stderr, and sends them to different files (and rotates them as necessary). It is currently hand-configureable (i.e. by altering some variables at the top of the script), and is more or less designed for use in our own environment. Tom Lane recently mentioned to me that a common complaint is that postgres doesn't have its own log rotator. There are, of course, plenty of good ones, and syslog itself works pretty well for most people. But there are still complaints from time to time about the lack of a "built in" log rotator. We'd be happy to release our rotator under the PostgreSQL BSD license, if it would be of use to people. I was thinking that perhaps contrib/ would be a good place for it, since the idea is to reduce complaints that there's no log rotator "included". Is anyone interested in having pglog-rotator? A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
Andrew Sullivan <andrew@libertyrms.info> writes:
> Is anyone interested in having pglog-rotator?
FWIW, I saw an early version of pglog-rotator about a year and a half
ago (while consulting for LibertyRMS), and thought at the time that
it was pretty cool. So I'm for including it ... maybe even as
mainstream instead of contrib.
regards, tom lane
Would the plan be to add it to pg_ctl? > Andrew Sullivan <andrew@libertyrms.info> writes: > > Is anyone interested in having pglog-rotator? > > FWIW, I saw an early version of pglog-rotator about a year and a half > ago (while consulting for LibertyRMS), and thought at the time that > it was pretty cool. So I'm for including it ... maybe even as > mainstream instead of contrib. > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly
"Jim Buttafuoco" <jim@contactbda.com> writes:
> Would the plan be to add it to pg_ctl?
You would not actually have to: you could just pipe pg_ctl's output to
pglog-rotator. But I think it'd be cool if pg_ctl had an option to use
pglog-rotator, or maybe even adopt it as standard behavior.
I think we would have to make the rotator script be mainstream rather
than contrib if we wanted pg_ctl to use it directly. That was why I was
thinking maybe mainstream ...
Andrew, could you toss up the script on pgsql-patches just so people can
take a look? Then we could think more about where to go with it.
regards, tom lane
Does this log rotator do something that apache's doesn't? Dave On Thu, 2003-04-03 at 13:41, Tom Lane wrote: > "Jim Buttafuoco" <jim@contactbda.com> writes: > > Would the plan be to add it to pg_ctl? > > You would not actually have to: you could just pipe pg_ctl's output to > pglog-rotator. But I think it'd be cool if pg_ctl had an option to use > pglog-rotator, or maybe even adopt it as standard behavior. > > I think we would have to make the rotator script be mainstream rather > than contrib if we wanted pg_ctl to use it directly. That was why I was > thinking maybe mainstream ... > > Andrew, could you toss up the script on pgsql-patches just so people can > take a look? Then we could think more about where to go with it. > > regards, tom lane > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org -- Dave Cramer <dave@fastcrypt.com> Cramer Consulting
On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote: > You would not actually have to: you could just pipe pg_ctl's output to > pglog-rotator. But I think it'd be cool if pg_ctl had an option to use > pglog-rotator, or maybe even adopt it as standard behavior. It's currently built to call a program, and read its stdout and stderr, rather than acting as a pipe. I guess it shouldn't be too hard to modify, though. We actually call the postmaster directly with it, so we use it as a replacement for pg_ctl at startup. > Andrew, could you toss up the script on pgsql-patches just so people can > take a look? Then we could think more about where to go with it. Ok, I sent it. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Thu, Apr 03, 2003 at 02:12:03PM -0500, Dave Cramer wrote: > Does this log rotator do something that apache's doesn't? Probably not. This was just easier for us. A little information might be handy here: we run postgres nder a hosted environment, and we do not have root on the relevant boxes. So installing anything even a little complicated means building everything ourselves. As a result, we end up re-creating plenty of functionality just to make it easy to install. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Thu, Apr 03, 2003 at 01:41:08PM -0500, Tom Lane wrote: > Andrew, could you toss up the script on pgsql-patches just so people can > take a look? Then we could think more about where to go with it. Ok, the first try failed (of course) because I wasn't subscribed. Should be there now, though. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
Andrew Sullivan writes: > Is anyone interested in having pglog-rotator? What would get me a whole lot more excited is if the server could write directly to a file and do its own rotating (or at least reopening of files). Considering that your rotator is tailored to a rather specific setup, it doesn't do anything better compared to established ones, it prevents the use of pg_ctl, it's written in Perl, and it doesn't do anything for Windows users, I think it's not suitable for a general audience. -- Peter Eisentraut peter_e@gmx.net
On Fri, 4 Apr 2003, Peter Eisentraut wrote: > Andrew Sullivan writes: > > > Is anyone interested in having pglog-rotator? > > What would get me a whole lot more excited is if the server could write > directly to a file and do its own rotating (or at least reopening of > files). > > Considering that your rotator is tailored to a rather specific setup, it > doesn't do anything better compared to established ones, it prevents the > use of pg_ctl, it's written in Perl, and it doesn't do anything for > Windows users, I think it's not suitable for a general audience. That said, a log rotation capability built right into pg_ctl or thereabouts would be a very nice feature. I.e. 'pg_ctl -r 86400 -l $PGDATA/logs/pgsql start' where -r is the rotation period in seconds. If it's an external program that pg_ctl calls that's fine, and it could even just be a carbon copy of apache's log rotater if their license is compatible (isn't it?)
On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote: > where -r is the rotation period in seconds. If it's an external program Ours rotates based on size rather than time. I can see some advantages to the time-based approach, but if you have wide variations in traffic, you run the risk of rotating over useful files with more or less empty ones if you use it. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Fri, Apr 04, 2003 at 05:13:13PM +0200, Peter Eisentraut wrote: > use of pg_ctl, it's written in Perl, and it doesn't do anything for > Windows users, I think it's not suitable for a general audience. It doesn't prevent the use of pg_ctl, although it does indeed prevent the use of pg_ctl for startup. I'm not sufficiently familiar with Windows to know how this does or does not help them. Could you elaborate? And what's wrong with Perl? A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Friday April 4 2003 9:16, scott.marlowe wrote: > > That said, a log rotation capability built right into pg_ctl or > thereabouts would be a very nice feature. I.e. 'pg_ctl -r 86400 -l > $PGDATA/logs/pgsql start' > > where -r is the rotation period in seconds. If it's an external program > that pg_ctl calls that's fine, and it could even just be a carbon copy of > apache's log rotater if their license is compatible (isn't it?) By way of feature ideas, one very convenient but not widely used feature of Apache's log rotator is the ability to specify a strftime() format string for the file extension. For example, if I want to have my logs rollover every 24 hours and be named log.Mon, log.Tue, log.Wed, I say something like pg_ctl start | rotatelogs 86400 "%a" This causes the logs to overwrite themselves every seven days, taking log maintenance time to very near zero. We also customized our use of it to allow us to automatically move existing logs out of the way to "log.1", "log.2", or to simply overwrite existing logs. Ed
On Friday April 4 2003 10:04, Ed L. wrote: > By way of feature ideas, one very convenient but not widely used feature > of Apache's log rotator is the ability to specify a strftime() format > string for the file extension. For example, if I want to have my logs > rollover every 24 hours and be named log.Mon, log.Tue, log.Wed, I say > something like > > pg_ctl start | rotatelogs 86400 "%a" More accurately, something like this: pg_ctl start | rotatelogs 86400 "log.%a" Ed
Peter Eisentraut wrote: > > Andrew Sullivan writes: > > > Is anyone interested in having pglog-rotator? > > What would get me a whole lot more excited is if the server could write > directly to a file and do its own rotating (or at least reopening of > files). From a technical point of view I don't think that is desirable. The entire log traffic would have to be routed through the postmaster, as it is in LibertyRMS's log rotator now through the perl script. And we really try to keep everything outside the postmaster that does not absolutely have to be in there for stability reasons. We can discuss if the log rotator should be a child process of the postmaster or the other way round, but that will not change the flow of bytes between the processes in any way. I would say it's better the way it is, because it does not pollute the postmasters wait logic with another exception. My ideal solution would be to integrate the log rotators functionality into a C version of pg_ctl that forks and detaches from the control terminal in the way, daemons should. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
On Fri, 4 Apr 2003, Andrew Sullivan wrote: > On Fri, Apr 04, 2003 at 09:16:39AM -0700, scott.marlowe wrote: > > where -r is the rotation period in seconds. If it's an external program > > Ours rotates based on size rather than time. I can see some > advantages to the time-based approach, but if you have wide > variations in traffic, you run the risk of rotating over useful files > with more or less empty ones if you use it. I would want time based for sure, and I can see the use for size based splitting as well. I wouldn't be hard to have it do both would it? I just like the idea of it being one of the dozens or so options for pg_ctl so it's painless to use for joe six pack. pg_ctl -r 86400 -l $PGDATA/logs/pgsql where -r is the rotation period OR pg_ctl -f 10M -l $PGDATA/logs/pgsql where -f is the max file size of a log I'd recommend that the nameing convnention should probably be: filenamespec.timestamp, like: $PGDATA/logs/pgsql.1049414400 for time rotated logs, and filename.incnumber like: $PGDATa/logs/pgsql.0000000001
Peter Eisentraut <peter_e@gmx.net> writes:
> What would get me a whole lot more excited is if the server could write
> directly to a file and do its own rotating (or at least reopening of
> files).
AFAICS, the only practical way to do this is to have a single process
collecting the stdout/stderr from the postmaster and all its children.
pglog-rotator is one implementation of that approach.
I too would rather this functionality were integrated into the server,
but I haven't noticed anyone stepping up to the plate to do it.
> Considering that your rotator is tailored to a rather specific setup, it
> doesn't do anything better compared to established ones, it prevents the
> use of pg_ctl, it's written in Perl, and it doesn't do anything for
> Windows users, I think it's not suitable for a general audience.
These might be good arguments for not putting it into the mainstream,
but I don't think they have any force if we consider it for contrib.
I feel we really ought to have *some* rotator included in the standard
distro, just so that the Admin Guide can point to a concrete solution
instead of having to arm-wave about what you can get off the net.
If someone can offer a better alternative than Andrew's, great, let's
see it.
regards, tom lane
On Friday April 4 2003 10:19, Tom Lane wrote: > > I feel we really ought to have *some* rotator included in the standard > distro, just so that the Admin Guide can point to a concrete solution > instead of having to arm-wave about what you can get off the net. > If someone can offer a better alternative than Andrew's, great, let's > see it. Out of curiosity, are there issues preventing inclusion of Apache's log rotation code? It seems you'd be hard-pressed to find a more battle-hardened log rotator. Obviously some people also wish to rotate based on log file size, so adding both to contrib at least seems sensible. Ed
On Fri, 4 Apr 2003, Ed L. wrote:
> On Friday April 4 2003 10:19, Tom Lane wrote:
> >
> > I feel we really ought to have *some* rotator included in the standard
> > distro, just so that the Admin Guide can point to a concrete solution
> > instead of having to arm-wave about what you can get off the net.
> > If someone can offer a better alternative than Andrew's, great, let's
> > see it.
>
> Out of curiosity, are there issues preventing inclusion of Apache's log
> rotation code? It seems you'd be hard-pressed to find a more
> battle-hardened log rotator.
>
> Obviously some people also wish to rotate based on log file size, so adding
> both to contrib at least seems sensible.
OK, I'm playing with the pg_ctl script that comes with 7.3, and trying to
make it startup with apaches rotatelog script, but this line won't pipe
output. I'm a total noob at bash shell scripting, so please feel free to
snicker when you answer.
rotatelogs is in my path and all, it just never sees it.
"$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> rotatelogs is in my path and all, it just never sees it.
You mean the command fails? Or just that it doesn't capture output?
> "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION 2>&1 &
Most if not all of the postmaster's log output goes to stderr, so you'd need
"$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
to have any hope of useful results.
regards, tom lane
On Friday April 4 2003 11:58, Tom Lane wrote:
> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > rotatelogs is in my path and all, it just never sees it.
>
> You mean the command fails? Or just that it doesn't capture output?
>
> > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > 2>&1 &
>
> Most if not all of the postmaster's log output goes to stderr, so you'd
> need
>
> "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
>
> to have any hope of useful results.
Hmmm. I would have agreed 2>&1 was needed, too, but this command seems to
routinely capture all output, including ERRORs:
nohup pg_ctl start | nohup rotatelogs server_log.%a 86400
Ed
On Fri, 4 Apr 2003, Tom Lane wrote: > "scott.marlowe" <scott.marlowe@ihs.com> writes: > > rotatelogs is in my path and all, it just never sees it. > > You mean the command fails? Or just that it doesn't capture output? The database starts, but rotatelogs doesn't get run. I.e. it's just like everything after the | symbol isn't there.
"Ed L." <pgsql@bluepolka.net> writes:
> Hmmm. I would have agreed 2>&1 was needed, too, but this command seems to
> routinely capture all output, including ERRORs:
> nohup pg_ctl start | nohup rotatelogs server_log.%a 86400
That's 'cause pg_ctl internally redirects the postmaster's stderr.
regards, tom lane
On Fri, 4 Apr 2003, Ed L. wrote:
> On Friday April 4 2003 11:58, Tom Lane wrote:
> > "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > > rotatelogs is in my path and all, it just never sees it.
> >
> > You mean the command fails? Or just that it doesn't capture output?
> >
> > > "$po_path" ${1+"$@"} </dev/null | $PGPATH/rotatelogs $logfile $DURATION
> > > 2>&1 &
> >
> > Most if not all of the postmaster's log output goes to stderr, so you'd
> > need
> >
> > "$po_path" ${1+"$@"} </dev/null 2>&1 | $PGPATH/rotatelogs ...
> >
> > to have any hope of useful results.
>
> Hmmm. I would have agreed 2>&1 was needed, too, but this command seems to
> routinely capture all output, including ERRORs:
>
> nohup pg_ctl start | nohup rotatelogs server_log.%a 86400
OK, So I tried putting the 2>&1 before the | and all. No matter what I
try, every from the | on is ignored. ps doesn't show it, and neither does
pg_ctl status. Both show a command line of
/usr/local/pgsql/bin/postmaster as the only input to start the server.
Now, the thing is, I've tried this with hardcoded values, like:
"$po_path" ${1+"$@"} </dev/null 2>&1 /usr/local/pgsql/bin/rotatelogs
/mnt/d1/data/logs/pglog 86400
where I know the logs directory exists. It works if I do:
pg_ctl start | rotatelogs $PGDATA/pglog 86400 2>1&
and puts the log files there.
I've copied rotatelogs into the /usr/local/pgsql/bin directory as well.
So, I'm thinking this is my weakness in shell scripting that's getting me
here, and that the shell is eating the |, not passing it out with the
postmaster to be used when it starts.
On Friday April 4 2003 2:17, scott.marlowe wrote: > > OK, So I tried putting the 2>&1 before the | and all. No matter what I > try, every from the | on is ignored. ps doesn't show it, and neither > does pg_ctl status. Both show a command line of > /usr/local/pgsql/bin/postmaster as the only input to start the server. Not clear if you're looking at it this way or if this is your problem, but you can't really tell there is log rotation going on just by grepping ps for postmaster because ps does not typically show the postmaster and the rotatelogs together on the same line. I wouldn't expect pg_ctl status to know anything at all about rotatelogs when you pipe it like this. Ed
On Fri, 4 Apr 2003, Ed L. wrote: > On Friday April 4 2003 2:17, scott.marlowe wrote: > > > > OK, So I tried putting the 2>&1 before the | and all. No matter what I > > try, every from the | on is ignored. ps doesn't show it, and neither > > does pg_ctl status. Both show a command line of > > /usr/local/pgsql/bin/postmaster as the only input to start the server. > > Not clear if you're looking at it this way or if this is your problem, but > you can't really tell there is log rotation going on just by grepping ps > for postmaster because ps does not typically show the postmaster and the > rotatelogs together on the same line. I wouldn't expect pg_ctl status to > know anything at all about rotatelogs when you pipe it like this. Hey, do you guys think that a setting of silent_mode = false might affect no log files getting created? I had it right as soon as I added Tom's recommended 2>&1 but spent another 30 minutes figuring out why my log file wasn't getting created / filled. Thanks for the help.
"scott.marlowe" <scott.marlowe@ihs.com> writes:
> Hey, do you guys think that a setting of silent_mode = false might affect
> no log files getting created?
No, but setting it to true would be bad news.
regards, tom lane
On Fri, 4 Apr 2003, Tom Lane wrote:
> "scott.marlowe" <scott.marlowe@ihs.com> writes:
> > Hey, do you guys think that a setting of silent_mode = false might affect
> > no log files getting created?
>
> No, but setting it to true would be bad news.
That's what I'd meant actually. I had to turn of silent mode... You know
you're having a bad day when your email explaining how stupid you are is
factually incorrect. :-)
If anyone wants the diff, here it is:
22c22
< $CMDNAME start [-w] [-D DATADIR] [-s] [-l FILENAME] [-o \"OPTIONS\"]
---
> $CMDNAME start [-w] [-D DATADIR] [-s] [-r DURATION] [-l FILENAME]
[-o \"OPTIONS\"]
39a40,41
> -r DURATION invoke log rotation with DURATION seconds
> between rotation of files.
155a158,161
> -r)
> DURATION="$2"
> shift
> ;;
336c342,346
< "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
---
> if [ -n "$DURATION" ]; then
> "$po_path" ${1+"$@"} </dev/null 2>&1| $PGPATH/rotatelogs
$logfile $DURATION 2>&1 &
> else
> "$po_path" ${1+"$@"} </dev/null >>$logfile 2>&1 &
> fi
Tom Lane writes: > AFAICS, the only practical way to do this is to have a single process > collecting the stdout/stderr from the postmaster and all its children. I think not. It's a little tricky handling it directly in the child processes, but it's been done before. > If someone can offer a better alternative than Andrew's, great, let's > see it. How about the attached one, which I floated a while ago but which didn't generate much interest. -- Peter Eisentraut peter_e@gmx.net ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> AFAICS, the only practical way to do this is to have a single process
>> collecting the stdout/stderr from the postmaster and all its children.
> I think not. It's a little tricky handling it directly in the child
> processes, but it's been done before.
A "little" tricky? Thanks, but no thanks ... for one thing, there'd be
no easy way to know when all the children had switched over to writing
the new file. Also, at least for not-too-long messages, writing on a
single pipe gives atomicity guarantees that AFAIK do not exist when
writing a file through multiple independently opened descriptors. In
the latter case I think we'd have lots of trouble with interleaving of
messages from different backends.
>> If someone can offer a better alternative than Andrew's, great, let's
>> see it.
> How about the attached one, which I floated a while ago but which didn't
> generate much interest.
Seems like a good bare-bones file writer; but how about all those
frammishes that people ask for like generating date-based filenames,
switching every so many bytes, etc? Also, it'd be nice not to be
dependent on a cron job to tickle the switchover.
I do think there's an efficiency argument for having the log writer
coded in C, so starting with what you have here and building up might
be a better idea than starting with Andrew's perl script. But the
important thing in my mind is to get something in there.
We should also take a look at Apache's rotator to see if there's any need
to reinvent the wheel at all. I have not seen it, am not even sure what
it's written in...
regards, tom lane
On Fri, 4 Apr 2003, Tom Lane wrote: > We should also take a look at Apache's rotator to see if there's any need > to reinvent the wheel at all. I have not seen it, am not even sure what > it's written in... It's written in 140 lines of C (blank lines and all), and has been very solid in my experience. I don't know of any deficiencies that would warrant rewriting it. Jon
Tom Lane writes: > Seems like a good bare-bones file writer; but how about all those > frammishes that people ask for like generating date-based filenames, > switching every so many bytes, etc? Also, it'd be nice not to be > dependent on a cron job to tickle the switchover. Linux systems have a standard system log rotation mechanism (see logrotate(8)), which can rotate logs by size and time and has a number of other features. I would rather depend on that kind of preferred system mechanism than rolling out our own. And we already depend on cron for vacuuming anyway. -- Peter Eisentraut peter_e@gmx.net
Tom Lane wrote:
> Seems like a good bare-bones file writer; but how about all those
> frammishes that people ask for like generating date-based filenames,
> switching every so many bytes, etc? Also, it'd be nice not to be
> dependent on a cron job to tickle the switchover.
One of the earlier times this discussion came up I wrote a log
rotation program too. It will rotate based on time, file size,
and/or SIGHUP:
ftp://ftp.nemeton.com.au/pub/src/logwrite-1.0alpha.tar.gz
Written in C, BSD license, used on my production systems. Needs to be
updated to not have 'alpha' in the URL ... no bug reports in 18 months
means that it could at least be beta. ;-) It should be portable: I've
built it on *BSD, HP-UX, and Linux at least. I'd need help for
Windows.
I took pains to deal with some of the concerns Tom raised last time
about not giving up and exiting when a filesystem fills up. This is
something that the Apache rotatelogs program didn't do at the time I
looked at it, else I'd have not written this at all.
> I too would rather this functionality were integrated into the server,
> but I haven't noticed anyone stepping up to the plate to do it.
Shouldn't be hard: the server only has to create a pipe, and run the
log rotation program. For extra robustness re-starting the log
rotation program if it exits ("don't kill -9 the log rotator") is a
good idea.
I'll look at doing this, unless the discussion heads somewhere else
e.g., if Peter decides to integrate his code. The server changes are
probably independent of whichever log rotation program is preferred
anyway.
Regards,
Giles
On Sun, Apr 06, 2003 at 11:03:27PM +0200, Peter Eisentraut wrote: > Linux systems have a standard system log rotation mechanism (see > logrotate(8)), which can rotate logs by size and time and has a number of > other features. I would rather depend on that kind of preferred system But it's not available on every platform. And according to my /usr/share/doc/logrotate/copyright, it's GPL, so you can't redistribute it with PostgreSQL. Building it in was one of the criteria, I thought > And we already depend on cron for vacuuming anyway. But that dependency is actually a liability, because there's no way to say, "Hey, I really need to be vacuumed now." We (at Liberty) are currently going through a remarkable number of hoops trying to get around that very limitation A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
Andrew Sullivan writes: > But it's not available on every platform. And according to my > /usr/share/doc/logrotate/copyright, it's GPL, so you can't > redistribute it with PostgreSQL. Building it in was one of the > criteria, I thought My point was that log file rotation should be left up to the system administrator. Look at other servers on your system (SMTP, DNS, whatever). How do they handle it? > > And we already depend on cron for vacuuming anyway. > > But that dependency is actually a liability, because there's no way > to say, "Hey, I really need to be vacuumed now." psql -c 'VACUUM' ? -- Peter Eisentraut peter_e@gmx.net
On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote: > > My point was that log file rotation should be left up to the system > administrator. Look at other servers on your system (SMTP, DNS, > whatever). How do they handle it? PostgreSQL is not a system process, and I think it's a mistake to assume that it is. We, for instance, do not have root on the machines we use. It's important to assume that users needn't be system administrators to use the system. I suppose, however, you could make the argument that log rotation should be the responisibility of the adminisistrator of the PostgreSQL server. But that just amounts to an argument that nothing needs to be done: as we see, there are lots of log management facilities on offer, and none of them are included with PostgreSQL. I don't feel strongly about which one to use. But since people frequently complain that the feature is not available in PostgreSQL, it does seem that, if it's not too much trouble, adding the feature is worth it. > > But that dependency is actually a liability, because there's no way > > to say, "Hey, I really need to be vacuumed now." > > psql -c 'VACUUM' ? I meant on the part of the back end. If you have a busy system on which some tables need very frequent vacuuming, but it gets unpredictable traffi, you don't just want to say, "Heck, let's vacuum every hour." You want to know _actually_ whether the table needs vacuuming. So you come up with a bunch of profiles, &c., and do a pile of work to figure out which tables really need attention. But that's not free: it takes cycles on the machine, and the hypotheical case is one in which the machine is already under heavy load. So cron is not really the answer. A -- ---- Andrew Sullivan 204-4141 Yonge Street Liberty RMS Toronto, Ontario Canada <andrew@libertyrms.info> M2P 2A8 +1 416 646 3304 x110
On Sun, 6 Apr 2003, Peter Eisentraut wrote: > Tom Lane writes: > > > Seems like a good bare-bones file writer; but how about all those > > frammishes that people ask for like generating date-based filenames, > > switching every so many bytes, etc? Also, it'd be nice not to be > > dependent on a cron job to tickle the switchover. > > Linux systems have a standard system log rotation mechanism (see > logrotate(8)), which can rotate logs by size and time and has a number of > other features. I would rather depend on that kind of preferred system > mechanism than rolling out our own. And we already depend on cron for > vacuuming anyway. How about we set up configure to check what we're on and what's available, (i.e. rotatelogs, logrotate, joesbiglogrotatorscript, etc...) and configure pg_ctl to use one of them? It's a good probability that most flavors of Unix have log rotators of some kind as a built in, and we can include a standard one as well. That way, if you want to use the same log rotator with postgresql as you use with the rest of your system, you can, and if you just want the built in one, you can use it, and if you don't want any log rotation, everything still works the same as before.
On Sunday 06 April 2003 18:54, Andrew Sullivan wrote: > On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote: > > My point was that log file rotation should be left up to the system > > administrator. Look at other servers on your system (SMTP, DNS, > > whatever). How do they handle it? > PostgreSQL is not a system process, and I think it's a mistake to > assume that it is. We, for instance, do not have root on the > machines we use. It's important to assume that users needn't be > system administrators to use the system. I personally believe that making the assumption that PostgreSQL is not a system process is wrong. One can run system services as a normal user (in fact, it is recommended that as few system services as is possible should run as root); but the fact that a daemon is running as a normal user doesn't make it not a system process. But that's just a difference of system administration opinion. However, I can see the utility of a bundled simple log rotator. The key word is simple -- we have the full-fledged route now, called syslog. And if someone needs a better logrotator they can certainly get one of the many that are already available. At the same time I don't necessarily want such a log rotator to be the default. We have syslog as the default. If someone has the particular need for a stderr/stdout log rotator, then let it be a configure option. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Andrew Sullivan writes: > PostgreSQL is not a system process, and I think it's a mistake to > assume that it is. The point is that PostgreSQL should fit nicely with the customs of the system that it runs on. This starts with the oft-discussed file system layout, the use of syslog in the first place, using 'cron' and 'at' instead of rolling our own mechanisms to schedule jobs, as is occasionally requested, fitting in with the startup scripts system, and so on. > I suppose, however, you could make the argument that log rotation > should be the responisibility of the adminisistrator of the > PostgreSQL server. But that just amounts to an argument that nothing > needs to be done: as we see, there are lots of log management > facilities on offer, and none of them are included with PostgreSQL. That is not the argument. What we need to do is to make it *possible* to rotate the logs without shutting down the server, not (necessarily) do the rotation ourselves. How can we even begin to do that? Do we need to invent a configuration language that can control when to rotate, where to move the old logs, when to delete the even older logs, etc.? > I meant on the part of the back end. If you have a busy system on > which some tables need very frequent vacuuming, but it gets > unpredictable traffi, you don't just want to say, "Heck, let's vacuum > every hour." You want to know _actually_ whether the table needs > vacuuming. That is an argument that manual vacuum is a liability, not the use of cron for it. -- Peter Eisentraut peter_e@gmx.net
Lamar Owen wrote: > On Sunday 06 April 2003 18:54, Andrew Sullivan wrote: > > On Mon, Apr 07, 2003 at 12:42:34AM +0200, Peter Eisentraut wrote: > > > My point was that log file rotation should be left up to the system > > > administrator. Look at other servers on your system (SMTP, DNS, > > > whatever). How do they handle it? > > > PostgreSQL is not a system process, and I think it's a mistake to > > assume that it is. We, for instance, do not have root on the > > machines we use. It's important to assume that users needn't be > > system administrators to use the system. > I personally believe that making the assumption that PostgreSQL is not > a system process is wrong. One can run system services as a normal > user (in fact, it is recommended that as few system services as is > possible should run as root); but the fact that a daemon is running as > a normal user doesn't make it not a system process. But that's just a > difference of system administration opinion. I think the mistake lies in making the "design" assumption that PostgreSQL is either one or the other. - There are contexts in which it forcibly is "systemy," such as when it is used for password authentication using somethinglike PAM. In that case, whatever userid it runs as, it's a forcible "system" dependancy. Users can't log in untilPostgreSQL is running. - There are contexts where it will run as a "part of the system," as is typically the case when someone uses "apt-get installpostgresql" or "rpm -i postgres*.rpm" - In a "hosted" environment, it may be unacceptable to, in any manner, treat PostgreSQL or any related services as "partof the system." cron obviously *is* a "part of the system," but if you're not the system administrator, you may have/no/ ability to connect in to "system" logging services. (In the environment where pgrotatelog runs, that is indeedthe case.) These are /all/ legitimate scenarios for PostgreSQL to be in use. --> Assuming PostgreSQL /is/ a system process is wrong. --> Assuming PostgreSQL /is not/ a system process is wrong. There are situations where either can be true, and it is vital for PostgreSQL to be able to support both. -- (concatenate 'string "cbbrowne" "@ntlug.org") http://www.ntlug.org/~cbbrowne/unix.html "There I was, lying, cheating and back-stabbing my way up the corporate ladder, feeling pretty darn good about myself, when someone told me the 'J' in 'WWJD' meant *Jesus* I thought it meant *Judas*! Hoo boy, am I red in the face!"
multilog andrew@libertyrms.info (Andrew Sullivan) wrote: > Since now is the time for contrib/ flamewars, this seemed a good time > to suggest this. > ... > > Is anyone interested in having pglog-rotator?
Tom Lane writes: > AFAICS, the only practical way to do this is to have a single process > collecting the stdout/stderr from the postmaster and all its children. I think not. It's a little tricky handling it directly in the child processes, but it's been done before. > If someone can offer a better alternative than Andrew's, great, let's > see it. How about the attached one, which I floated a while ago but which didn't generate much interest. -- Peter Eisentraut peter_e@gmx.net ---------------------------(end of broadcast)--------------------------- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
Peter Eisentraut writes:
> I think not. It's a little tricky handling it directly in the child
> processes, but it's been done before.
Certainly has...
In the cfengine2 code base, the relevant file is "rotate.c"; it
essentially attaches the file descriptor to a new file "on the fly," and
does so as a separate process. The code apparently works on NT, too;
there is a comment that indicates that they use chown() rather than
fchown() because the latter doesn't exist on NT.
In fact, that file is well worth taking a look at for strategies on
this. It's GPLed code, and so may not be suitable for integration, but
there are doubtless some useful techniques to be seen...
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','cbbrowne.com').
http://cbbrowne.com/info/languages.html
"Microsoft is sort of a mixture between the Borg and the
Ferengi. Combine the Borg marketing with Ferengi networking..."
-- Andre Beck in dcouln