Thread: BUG #5741: syslog line length
The following bug has been logged online: Bug reference: 5741 Logged by: heasley Email address: heas@shrubbery.net PostgreSQL version: 8.4 Operating system: solaris Description: syslog line length Details: * Max string length to send to syslog(). Note that this doesn't count the * sequence-number prefix we add, and of course it doesn't count the prefix * added by syslog itself. On many implementations it seems that the hard * limit is approximately 2K bytes including both those prefixes. */ #ifndef PG_SYSLOG_LIMIT #define PG_SYSLOG_LIMIT 1024 #endif solaris' syslogd limits the line length to 1024, with a FQDN and it's silly "msg ID" quite a bit is dropped by syslogd.
On 02/11/2010 17:17, heasley wrote: > The following bug has been logged online: > > Bug reference: 5741 > Logged by: heasley > Email address: heas@shrubbery.net > PostgreSQL version: 8.4 > Operating system: solaris > Description: syslog line length > Details: > > * Max string length to send to syslog(). Note that this doesn't count the > * sequence-number prefix we add, and of course it doesn't count the prefix > * added by syslog itself. On many implementations it seems that the > hard > * limit is approximately 2K bytes including both those prefixes. > */ > #ifndef PG_SYSLOG_LIMIT > #define PG_SYSLOG_LIMIT 1024 > #endif > > solaris' syslogd limits the line length to 1024, with a > FQDN and it's silly "msg ID" quite a bit is dropped by > syslogd. > I've been having the exact same problem with CentOS 5.5 these days (and a customer's Red Hat 5.4). This same problem occurs with sysklogd, which has a #define MAXLINE 1024 /* maximum line length */ Replacing sysklogd with rsyslog, metalog or syslog-ng solves the problem, as they all have 2048 for their buffer size. But CentOS and RedHat 5 both seem to have sysklogd as the default logger. I'll solve this by changing their logger, but I thought it would be worthy to mention.
On Tue, Nov 2, 2010 at 12:17 PM, heasley <heas@shrubbery.net> wrote: > > The following bug has been logged online: > > Bug reference: =A0 =A0 =A05741 > Logged by: =A0 =A0 =A0 =A0 =A0heasley > Email address: =A0 =A0 =A0heas@shrubbery.net > PostgreSQL version: 8.4 > Operating system: =A0 solaris > Description: =A0 =A0 =A0 =A0syslog line length > Details: > > * Max string length to send to syslog(). =A0Note that this doesn't count = the > =A0* sequence-number prefix we add, and of course it doesn't count the pr= efix > =A0* added by syslog itself. =A0 =A0 =A0On many implementations it seems = that the > hard > =A0* limit is approximately 2K bytes including both those prefixes. > =A0*/ > #ifndef PG_SYSLOG_LIMIT > #define PG_SYSLOG_LIMIT 1024 > #endif > > solaris' syslogd limits the line length to 1024, with a > FQDN and it's silly "msg ID" quite a bit is dropped by > syslogd. You can recompile with -DPG_SYSLOG_LIMIT=3D<some smaller value>. We could add a configuration setting for this, if it's affecting a lot of people. --=20 Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Wed, Jun 29, 2011 at 01:49:05PM -0400, Robert Haas:
> You can recompile with -DPG_SYSLOG_LIMIT=<some smaller value>.
>
> We could add a configuration setting for this, if it's affecting a lot
> of people.
that'd be useful. we have recently decided to abandon oracle's lovely support
{de-}structure; so i've lost interest. but, thanks for the suggested
work-around.
On Fri, Jun 10, 2011 at 04:21:59PM +0200, Marc Cousin wrote: > On 02/11/2010 17:17, heasley wrote: > > The following bug has been logged online: > > > > Bug reference: 5741 > > Logged by: heasley > > Email address: heas@shrubbery.net > > PostgreSQL version: 8.4 > > Operating system: solaris > > Description: syslog line length > > Details: > > > > * Max string length to send to syslog(). Note that this doesn't count the > > * sequence-number prefix we add, and of course it doesn't count the prefix > > * added by syslog itself. On many implementations it seems that the > > hard > > * limit is approximately 2K bytes including both those prefixes. > > */ > > #ifndef PG_SYSLOG_LIMIT > > #define PG_SYSLOG_LIMIT 1024 > > #endif > > > > solaris' syslogd limits the line length to 1024, with a > > FQDN and it's silly "msg ID" quite a bit is dropped by > > syslogd. > > > I've been having the exact same problem with CentOS 5.5 these days (and > a customer's Red Hat 5.4). > > This same problem occurs with sysklogd, which has a > #define MAXLINE 1024 /* maximum line length */ A PG_SYSLOG_LIMIT value that loses data to truncation on nearly every default GNU/Linux installation makes for a poor default. It seems we raised it from 128 to 1024 for version 8.4, on performance grounds: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=6b7eebc05e3bfd86757f5aa1f05bca463e92d7db How about 896? I reach that figure based on 1024 as a reasonable limit for the total syslog line, including prefix, and these estimates for the prefix parts: 25 - fixed-length content 16 - syslog_ident [default is "postgres"] 10 - pid 32 - hostname [certainly not a limit, but frequently adequate] 10 - sequence number [ditto] 3 - chunk number [ditto] 32 - facility/priority/message ID added by some syslog implementations = 128 Perhaps, though, the default should be outright conservative, like 512. We could also have a compile-time fixed overhead and factor in actual lengths of parts we know at runtime, but that smells like overkill. Opinions? -- Noah Misch http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Noah Misch <noah@2ndQuadrant.com> writes:
> A PG_SYSLOG_LIMIT value that loses data to truncation on nearly every default
> GNU/Linux installation makes for a poor default.
When I changed it, it was on the strength of tests showing that Fedora
8's default syslogger had a limit of 2KB. A quick check shows this is
still the case in Fedora 15. So I'm not convinced that "nearly every"
is anywhere near true.
> How about 896?
But having said that, I could go with 896 (or perhaps 900, which seems
a shade less weird).
regards, tom lane
On Fri, Aug 05, 2011 at 06:46:21PM -0400, Tom Lane wrote: > Noah Misch <noah@2ndQuadrant.com> writes: > > A PG_SYSLOG_LIMIT value that loses data to truncation on nearly every default > > GNU/Linux installation makes for a poor default. > > When I changed it, it was on the strength of tests showing that Fedora > 8's default syslogger had a limit of 2KB. A quick check shows this is > still the case in Fedora 15. So I'm not convinced that "nearly every" > is anywhere near true. Point taken. Marc hit the 1024 limit in RHEL 5, which appears to be based on Fedora 6. Perhaps Fedora began to patch in the higher limit in 7 or 8. I also observed the 1024 limit in Ubuntu 8.04 LTS. It is the limit in the latest source distributions of sysklogd, too. > > How about 896? > > But having said that, I could go with 896 (or perhaps 900, which seems > a shade less weird). That works for me. -- Noah Misch http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Attachment
Noah Misch <noah@2ndQuadrant.com> writes:
> On Fri, Aug 05, 2011 at 06:46:21PM -0400, Tom Lane wrote:
>> But having said that, I could go with 896 (or perhaps 900, which seems
>> a shade less weird).
> That works for me.
OK, applied to HEAD and 9.1.
regards, tom lane
Hi, On Sat, Aug 6, 2011 at 12:18 AM, Noah Misch <noah@2ndquadrant.com> wrote: > A PG_SYSLOG_LIMIT value that loses data to truncation on nearly every default > GNU/Linux installation makes for a poor default. On Sat, Aug 6, 2011 at 3:03 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > OK, applied to HEAD and 9.1. We hit this problem a few days ago on a server with RHEL 5 + PostgreSQL 8.4 + syslog. It made it quite difficult to work on the slow queries paralyzing the server as several of them had truncated lines in the middle of them. Any chance to have this fix backported to 8.4 and 9.0 as this is a regression introduced in 8.4? Thanks for your feedback. -- Guillaume