Thread: [7.0.2] rotating log files ...
can someone recommend on how to rotate the following log files without
having to stop/start the server?
postgres 85333 pgsql 1w VREG 13,131084 995 762037 /pgsql/logs/postmaster.5432.61308
postgres 85333 pgsql 2w VREG 13,131084 316501241 762038 /pgsql/logs/5432.61308
its the 316meg one and growing that I'm more interested in doing, obviously ...
I'm starting my postmaster as:
#!/bin/tcsh
setenv PORT 5432
setenv POSTMASTER /pgsql/bin/postmaster
unlimit
${POSTMASTER} -B 384 -N 192 \
-o "-F -o /pgsql/logs/${PORT}.$$ -S 32768" \
-i -p ${PORT} -D/pgsql/data >& /pgsql/logs/postmaster.${PORT}.$$ &
where I was hoping that the '-o /pgsql/logs/${PORT}.$$' would create a
'per postgres process' log file, but $$ is taking on the PID of the parent
process, not the children ...
suggestions?
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
Can you move the logfile then touch the original file to create an empty
one?
*shrug* just an idea :-)
-Mitch
----- Original Message -----
From: "The Hermit Hacker" <scrappy@hub.org>
To: <pgsql-admin@postgresql.org>
Sent: Monday, August 28, 2000 10:59 AM
Subject: [ADMIN] [7.0.2] rotating log files ...
>
> can someone recommend on how to rotate the following log files without
> having to stop/start the server?
>
> postgres 85333 pgsql 1w VREG 13,131084 995 762037
/pgsql/logs/postmaster.5432.61308
> postgres 85333 pgsql 2w VREG 13,131084 316501241 762038
/pgsql/logs/5432.61308
>
> its the 316meg one and growing that I'm more interested in doing,
obviously ...
>
> I'm starting my postmaster as:
>
> #!/bin/tcsh
> setenv PORT 5432
> setenv POSTMASTER /pgsql/bin/postmaster
> unlimit
> ${POSTMASTER} -B 384 -N 192 \
> -o "-F -o /pgsql/logs/${PORT}.$$ -S 32768" \
> -i -p ${PORT} -D/pgsql/data >&
/pgsql/logs/postmaster.${PORT}.$$ &
>
> where I was hoping that the '-o /pgsql/logs/${PORT}.$$' would create a
> 'per postgres process' log file, but $$ is taking on the PID of the parent
> process, not the children ...
>
> suggestions?
>
> Marc G. Fournier ICQ#7615664 IRC Nick:
Scrappy
> Systems Administrator @ hub.org
> primary: scrappy@hub.org secondary:
scrappy@{freebsd|postgresql}.org
>
>
The Hermit Hacker wrote: [snipped log stuff] > suggestions? Change to run with syslog. As of 7.0.2, syslog is working quite well -- and syslog targets are easily rollable. Since I've switched to syslog here, I haven't had any troubles with logs. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
* The Hermit Hacker <scrappy@hub.org> [000828 11:19] wrote: > > can someone recommend on how to rotate the following log files without > having to stop/start the server? > > postgres 85333 pgsql 1w VREG 13,131084 995 762037 /pgsql/logs/postmaster.5432.61308 > postgres 85333 pgsql 2w VREG 13,131084 316501241 762038 /pgsql/logs/5432.61308 > > its the 316meg one and growing that I'm more interested in doing, obviously ... This may not work: make a copy of it and try running a program that calls ftruncate() on the file. I think postgresql needs some utility to do this, I may whip one up when time allows, it should be trivial. -Alfred
* Mitch Vincent <mitch@venux.net> [000828 11:28] wrote: > Can you move the logfile then touch the original file to create an empty > one? > > *shrug* just an idea :-) that won't work, the shell has attached the process's stdout/err to the file, you can even remove the file and the server will still have an open reference to it and the space won't be reclaimed. -Alfred
Yeah, I thought about that after I sent the email. What can I say, I think with my index finger sometimes instead of my brain :-) -Mitch ----- Original Message ----- From: "Alfred Perlstein" <bright@wintelcom.net> To: "Mitch Vincent" <mitch@venux.net> Cc: "The Hermit Hacker" <scrappy@hub.org>; <pgsql-admin@postgresql.org> Sent: Monday, August 28, 2000 11:30 AM Subject: Re: [ADMIN] [7.0.2] rotating log files ... > * Mitch Vincent <mitch@venux.net> [000828 11:28] wrote: > > Can you move the logfile then touch the original file to create an empty > > one? > > > > *shrug* just an idea :-) > > that won't work, the shell has attached the process's stdout/err to > the file, you can even remove the file and the server will still have > an open reference to it and the space won't be reclaimed. > > -Alfred >
It's not normally a good idea to move/remove files that a process has an
open file handle for. The safest way is to copy the file to a new
location/name then cat /dev/null > logfile
Chris
Mitch Vincent wrote:
>
> Can you move the logfile then touch the original file to create an empty
> one?
>
> *shrug* just an idea :-)
>
> -Mitch
>
> ----- Original Message -----
> From: "The Hermit Hacker" <scrappy@hub.org>
> To: <pgsql-admin@postgresql.org>
> Sent: Monday, August 28, 2000 10:59 AM
> Subject: [ADMIN] [7.0.2] rotating log files ...
>
> >
> > can someone recommend on how to rotate the following log files without
> > having to stop/start the server?
> >
> > postgres 85333 pgsql 1w VREG 13,131084 995 762037
> /pgsql/logs/postmaster.5432.61308
> > postgres 85333 pgsql 2w VREG 13,131084 316501241 762038
> /pgsql/logs/5432.61308
> >
> > its the 316meg one and growing that I'm more interested in doing,
> obviously ...
> >
> > I'm starting my postmaster as:
> >
> > #!/bin/tcsh
> > setenv PORT 5432
> > setenv POSTMASTER /pgsql/bin/postmaster
> > unlimit
> > ${POSTMASTER} -B 384 -N 192 \
> > -o "-F -o /pgsql/logs/${PORT}.$$ -S 32768" \
> > -i -p ${PORT} -D/pgsql/data >&
> /pgsql/logs/postmaster.${PORT}.$$ &
> >
> > where I was hoping that the '-o /pgsql/logs/${PORT}.$$' would create a
> > 'per postgres process' log file, but $$ is taking on the PID of the parent
> > process, not the children ...
> >
> > suggestions?
> >
> > Marc G. Fournier ICQ#7615664 IRC Nick:
> Scrappy
> > Systems Administrator @ hub.org
> > primary: scrappy@hub.org secondary:
> scrappy@{freebsd|postgresql}.org
> >
> >
That won't work in and of itself since the filehandle to the open
file remains. Possibly do that and HUP the owning process.
I've no idea what happens when that particular process gets SIGHUP
(just because I have the source doesn't mean I read it for no reason)
On Mon, 28 Aug 2000, Mitch Vincent wrote:
> Can you move the logfile then touch the original file to create an empty
> one?
>
> *shrug* just an idea :-)
>
> -Mitch
>
> ----- Original Message -----
> From: "The Hermit Hacker" <scrappy@hub.org>
> To: <pgsql-admin@postgresql.org>
> Sent: Monday, August 28, 2000 10:59 AM
> Subject: [ADMIN] [7.0.2] rotating log files ...
>
>
> >
> > can someone recommend on how to rotate the following log files without
> > having to stop/start the server?
> >
> > postgres 85333 pgsql 1w VREG 13,131084 995 762037
> /pgsql/logs/postmaster.5432.61308
> > postgres 85333 pgsql 2w VREG 13,131084 316501241 762038
> /pgsql/logs/5432.61308
> >
> > its the 316meg one and growing that I'm more interested in doing,
> obviously ...
> >
> > I'm starting my postmaster as:
> >
> > #!/bin/tcsh
> > setenv PORT 5432
> > setenv POSTMASTER /pgsql/bin/postmaster
> > unlimit
> > ${POSTMASTER} -B 384 -N 192 \
> > -o "-F -o /pgsql/logs/${PORT}.$$ -S 32768" \
> > -i -p ${PORT} -D/pgsql/data >&
> /pgsql/logs/postmaster.${PORT}.$$ &
> >
> > where I was hoping that the '-o /pgsql/logs/${PORT}.$$' would create a
> > 'per postgres process' log file, but $$ is taking on the PID of the parent
> > process, not the children ...
> >
> > suggestions?
> >
> > Marc G. Fournier ICQ#7615664 IRC Nick:
> Scrappy
> > Systems Administrator @ hub.org
> > primary: scrappy@hub.org secondary:
> scrappy@{freebsd|postgresql}.org
> >
> >
>
On Mon, Aug 28, 2000 at 02:31:29PM -0400, Chris Ryan wrote: > It's not normally a good idea to move/remove files that a process has an > open file handle for. The safest way is to copy the file to a new > location/name then cat /dev/null > logfile What's wrong with moving the file when it's in use? Copying the file will take much longer, and you might loose log-entries that are written after cp but before truncate. -- Ragnar Kjorstad
*falls over in astonishment, gagging* *switch to CS professor mode* Filesystem corruption result highly likely. Reason: while you are touching the file, so is the process. It has a file handle open. Actually should be locked and it would be best if mv would honor that. Under other operating systems such as VMS, the file would be locked only SYSADMIN could mess with it. Think -- Linux is a Unix variant, a true multitasking multiprocessing operating (with thread (lightweight process) support, no less) system. While you are typing your mv command the process is running unless you suspend it somehow. Therefore it is busy writing to disk while you mess with its file. Even if you catch between writes, there is the write cache / ouptut buffers (which is why a close is often preceded by a fflush). REally you need to take some basic operating systems course (or at least read a book on the subject) if you do not understand. I will soon have my recommended books list going and will have a book on o/s there... On Mon, 28 Aug 2000, [iso-8859-1] Ragnar Kj�rstad wrote: > On Mon, Aug 28, 2000 at 02:31:29PM -0400, Chris Ryan wrote: > > It's not normally a good idea to move/remove files that a process has an > > open file handle for. The safest way is to copy the file to a new > > location/name then cat /dev/null > logfile > > What's wrong with moving the file when it's in use? > > Copying the file will take much longer, and you might loose log-entries > that are written after cp but before truncate. > > >
> > On Mon, Aug 28, 2000 at 02:31:29PM -0400, Chris Ryan wrote:
> > > It's not normally a good idea to move/remove files that a process has an
> > > open file handle for. The safest way is to copy the file to a new
> > > location/name then cat /dev/null > logfile
> >
> > What's wrong with moving the file when it's in use?
> >
> > Copying the file will take much longer, and you might loose log-entries
> > that are written after cp but before truncate.
* Dana Hudes <dhudes@hudes.org> [000828 13:07] wrote:
> *falls over in astonishment, gagging*
> *switch to CS professor mode*
>
> Filesystem corruption result highly likely.
> Reason: while you are touching the file, so is the process.
> It has a file handle open. Actually should be locked and it would
> be best if mv would honor that.
> Under other operating systems such as VMS, the file would be locked
> only SYSADMIN could mess with it.
> Think -- Linux is a Unix variant, a true multitasking
> multiprocessing operating (with thread (lightweight process) support, no
> less) system. While you are typing your mv command the process is running
> unless you suspend it somehow. Therefore it is busy writing to disk
> while you mess with its file. Even if you catch between writes, there is
> the write cache / ouptut buffers (which is why a close is often preceded
> by a fflush).
This is half wrong, if the rename operation happens on the same
filesystem then there is no chance of corruption. The OS will take
care of this. However if it's a mv to another filesystem then what
you're saying is a possibility.
I'm not sure if you miss-parsed Chris Ryan's question which I take
as a rename within the same filesystem, but I would not be so eager
to jump down his throat if you're not completely familiar with the
OS internals or not sure of the question.
(ponders how trivial it might be to write a program that swaps)
. o O (out open files out from under a running program)
This is a good way of fixing the problem:
cp logfile someplaceelse
killall -STOP postresql
# append any extra data written to the log
truncate -s 0 logfile # FreeBSD provides this as a userland tool
# you may have to write your own to call the OS's
# truncate/ftruncate routine
# or simply : cat /dev/null > logfile (my work)
killall -CONT postresql
you shouldn't loose any data that way.
--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."
The process still has an open file handle, and will continue to do so even after you move it. So, if your file is /var/log/messages, and you do a mv /var/log/messages /var/log/messages.old or something (I know that's stupid, but this is an example), the process will continue to write to /var/log/messages.old. The best way is what Chris said: copy the file, and cat /dev/null > logfile. That'll truncate it well. -----Original Message----- From: Ragnar Kjørstad [mailto:postgres@ragnark.vestdata.no] What's wrong with moving the file when it's in use? Copying the file will take much longer, and you might loose log-entries that are written after cp but before truncate. -- Ragnar Kjorstad
Just as an FYI ... I just enabled the syslog facility so that it all goes
through there instead ...
On Mon, 28 Aug 2000, Neil Toronto wrote:
> The process still has an open file handle, and will continue to do so even
> after you move it. So, if your file is /var/log/messages, and you do a mv
> /var/log/messages /var/log/messages.old or something (I know that's stupid,
> but this is an example), the process will continue to write to
> /var/log/messages.old.
>
> The best way is what Chris said: copy the file, and cat /dev/null > logfile.
> That'll truncate it well.
>
> -----Original Message-----
> From: Ragnar Kj�rstad [mailto:postgres@ragnark.vestdata.no]
>
> What's wrong with moving the file when it's in use?
>
> Copying the file will take much longer, and you might loose log-entries
> that are written after cp but before truncate.
>
>
> --
> Ragnar Kjorstad
>
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
On Mon, Aug 28, 2000 at 03:53:11PM -0400, Dana Hudes wrote: > Filesystem corruption result highly likely. > Reason: while you are touching the file, so is the process. > It has a file handle open. Actually should be locked and it would > be best if mv would honor that. > Under other operating systems such as VMS, the file would be locked > only SYSADMIN could mess with it. > Think -- Linux is a Unix variant, a true multitasking > multiprocessing operating (with thread (lightweight process) support, no > less) system. While you are typing your mv command the process is running > unless you suspend it somehow. Therefore it is busy writing to disk > while you mess with its file. Even if you catch between writes, there is > the write cache / ouptut buffers (which is why a close is often preceded > by a fflush). I think you may have gotten this wrong... 1. On linux (and most other unixes, I believe), the file-operations (like read, write and so on) are not tied with the filenames at all. They all work on an open filedescriptor, wich is linked to an inode. (via a dentry, but I don't think that's rellevant). If the filename is changed, or even removed, they don't care. Actually, if you look at how tmpfiles are implemented, you'll find that in most systems they do (open, unlink) and then go on working with a file. 2. rename is guaranteed to be atomic by posix, if I'm not mistaken. The problem you're probably refering to is when deleted files stay open and the system crashes. On non-journaling filesystems the blocks will not be freed until fsck is run. A few filesystems have had bugs that prevented the space from beeing freed up again - I'm not sure if theese problems were only caused when the system crashed or also under normal usage. > REally you need to take some basic operating systems course (or at least > read a book on the subject) if you do not understand. Actually I have, and I'we been working with linux-kernel development for some time (e.g. the reiserfs filesystem for linux) so I believe I have some idea of what I'm talking about. I'm not too arrogant to learn though, so let me know if there are errors in my argument. -- Ragnar Kjorstad
On Mon, Aug 28, 2000 at 02:37:24PM -0600, Neil Toronto wrote: > The process still has an open file handle, and will continue to do so even > after you move it. So, if your file is /var/log/messages, and you do a mv > /var/log/messages /var/log/messages.old or something (I know that's stupid, > but this is an example), the process will continue to write to > /var/log/messages.old. Naturally, you need to HUP the process when you're done moving the file. (and ofcourse, you need to have the rotated version on the same filesystem) > The best way is what Chris said: copy the file, and cat /dev/null > logfile. > That'll truncate it well. Copying the file will take much longer, and you might loose log-entries that are written after cp but before truncate. -- Ragnar Kjorstad
Hi,
Apart from using syslog, which apparently work fine (?) with
7.0.2, the 'rotatelogs' utility from apache's support dir is
great (No need to hack one up, as Alfred Perlstein suggest;
this babe been around and is quite stable).
Of course, you need to stop the server once, but I guess other
ways have been thouroughly dicussed in already :)
Enclosed is a Bournce shell script showing how I start postgres
on developers' machines [debug on] (run as root, will su pgsql)
On syslog... is syslog support reliable now? If so,
admin/pg-options.htm should be updated:
...Unfortunately many functions call directly printf() to print
their messages to stdout or stderr and this output can't be
redirected to syslog or have timestamps in it....
Cheers,
Palle
The Hermit Hacker wrote:
>
> can someone recommend on how to rotate the following log files without
> having to stop/start the server?
...
> suggestions?
>
> Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
> Systems Administrator @ hub.org
> primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
Attachment
> Hi, > > Apart from using syslog, which apparently work fine (?) with > 7.0.2, the 'rotatelogs' utility from apache's support dir is > great (No need to hack one up, as Alfred Perlstein suggest; > this babe been around and is quite stable). > > Of course, you need to stop the server once, but I guess other > ways have been thouroughly dicussed in already :) > > Enclosed is a Bournce shell script showing how I start postgres > on developers' machines [debug on] (run as root, will su pgsql) > > On syslog... is syslog support reliable now? If so, > admin/pg-options.htm should be updated: > > ...Unfortunately many functions call directly printf() to print > their messages to stdout or stderr and this output can't be > redirected to syslog or have timestamps in it.... Would someone be interested in converting our fprintf(stderr,...) calls to elog and submitting a patch, please? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026