Thread: Re: [ADMIN] 9.5 new setting "cluster name" and logging

Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Andres Freund
Date:
Hi,

(x-posting to -hackers, more relevant audience)

On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
> Now that there is a setting to give a cluster a "name", it would be nice to
> have an escape sequence in the log_line_prefix setting that could reference
> the cluster_name.

I've argued[1][2] for this when cluster_name was introduced, but back
then I seemed to have been the only one arguing for it. Josh later
jumped that train.

Given that we now had a number of people wishing for this, can we maybe
reconsider?

Greetings,

Andres Freund

[1] http://archives.postgresql.org/message-id/CADLWmXUm%3D7Y3UORZnGMdSC6p1eymO0k0JkH4NKr4KstdWk0P7g%40mail.gmail.com
[2] http://archives.postgresql.org/message-id/20140505101007.GU12715@awork2.anarazel.de


Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Joe Conway
Date:
On 02/08/2016 06:24 AM, Andres Freund wrote:
> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
>> Now that there is a setting to give a cluster a "name", it would be nice to
>> have an escape sequence in the log_line_prefix setting that could reference
>> the cluster_name.
>
> I've argued[1][2] for this when cluster_name was introduced, but back
> then I seemed to have been the only one arguing for it. Josh later
> jumped that train.
>
> Given that we now had a number of people wishing for this, can we maybe
> reconsider?

Seems like a reasonable idea to me. But if we add a log_line_prefix
setting, shouldn't we also add it to csvlog output too?

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development


Attachment

Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Thomas Munro
Date:
On Tue, Feb 9, 2016 at 5:30 AM, Joe Conway <mail@joeconway.com> wrote:
> On 02/08/2016 06:24 AM, Andres Freund wrote:
>> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
>>> Now that there is a setting to give a cluster a "name", it would be nice to
>>> have an escape sequence in the log_line_prefix setting that could reference
>>> the cluster_name.
>>
>> I've argued[1][2] for this when cluster_name was introduced, but back
>> then I seemed to have been the only one arguing for it. Josh later
>> jumped that train.
>>
>> Given that we now had a number of people wishing for this, can we maybe
>> reconsider?
>
> Seems like a reasonable idea to me. But if we add a log_line_prefix
> setting, shouldn't we also add it to csvlog output too?

Here's a tiny patch adding support for %C to log_line_prefix (this was
part of the cluster_name patch that didn't go it).

Given that csvlog's output format is hardcoded in write_csvlog, how is
it supposed to evolve without upsetting consumers of this data?
Wouldn't we first need to add a GUC that lets you control the columns
it outputs?

--
Thomas Munro
http://www.enterprisedb.com

Attachment

Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Stephen Frost
Date:
Thomas,

* Thomas Munro (thomas.munro@enterprisedb.com) wrote:
> On Tue, Feb 9, 2016 at 5:30 AM, Joe Conway <mail@joeconway.com> wrote:
> > On 02/08/2016 06:24 AM, Andres Freund wrote:
> >> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
> >>> Now that there is a setting to give a cluster a "name", it would be nice to
> >>> have an escape sequence in the log_line_prefix setting that could reference
> >>> the cluster_name.
> >>
> >> I've argued[1][2] for this when cluster_name was introduced, but back
> >> then I seemed to have been the only one arguing for it. Josh later
> >> jumped that train.
> >>
> >> Given that we now had a number of people wishing for this, can we maybe
> >> reconsider?
> >
> > Seems like a reasonable idea to me. But if we add a log_line_prefix
> > setting, shouldn't we also add it to csvlog output too?
>
> Here's a tiny patch adding support for %C to log_line_prefix (this was
> part of the cluster_name patch that didn't go it).
>
> Given that csvlog's output format is hardcoded in write_csvlog, how is
> it supposed to evolve without upsetting consumers of this data?
> Wouldn't we first need to add a GUC that lets you control the columns
> it outputs?

Not sure if you really want to go there, but I do agree with you and
there's a thread from a few years back about something similar:

http://www.postgresql.org/message-id/flat/20110112142345.GA4933@tamriel.snowman.net

Included in that thread is a patch, which likely requires some dusting
off, to add exactly that ability.

Thanks!

Stephen

Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Thomas Munro
Date:
On Tue, Feb 9, 2016 at 3:24 AM, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> (x-posting to -hackers, more relevant audience)
>
> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
>> Now that there is a setting to give a cluster a "name", it would be nice to
>> have an escape sequence in the log_line_prefix setting that could reference
>> the cluster_name.
>
> I've argued[1][2] for this when cluster_name was introduced, but back
> then I seemed to have been the only one arguing for it. Josh later
> jumped that train.
>
> Given that we now had a number of people wishing for this, can we maybe
> reconsider?

+1

I missed the part of this conversation that took place on -admin.  I
agree with Evan Rempel's post over there[1] that it's useful for
syslog users, and that it's not ideal to have to hijack syslog_ident
or explicitly copy the name into log_line_prefix.

[1] https://www.postgresql.org/message-id/56C64017.7050303%40uvic.ca

--
Thomas Munro
http://www.enterprisedb.com


Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Stephen Frost
Date:
* Thomas Munro (thomas.munro@enterprisedb.com) wrote:
> On Tue, Feb 9, 2016 at 3:24 AM, Andres Freund <andres@anarazel.de> wrote:
> > (x-posting to -hackers, more relevant audience)
> >
> > On 2016-01-29 22:19:45 -0800, Evan Rempel wrote:
> >> Now that there is a setting to give a cluster a "name", it would be nice to
> >> have an escape sequence in the log_line_prefix setting that could reference
> >> the cluster_name.
> >
> > I've argued[1][2] for this when cluster_name was introduced, but back
> > then I seemed to have been the only one arguing for it. Josh later
> > jumped that train.
> >
> > Given that we now had a number of people wishing for this, can we maybe
> > reconsider?
>
> +1
>
> I missed the part of this conversation that took place on -admin.  I
> agree with Evan Rempel's post over there[1] that it's useful for
> syslog users, and that it's not ideal to have to hijack syslog_ident
> or explicitly copy the name into log_line_prefix.
>
> [1] https://www.postgresql.org/message-id/56C64017.7050303%40uvic.ca

I'm with Thomas on this and I disagree that the "csvlog bloat" argument
has merit.  If we're worried about bloat in csv then we should provide a
way for users to control what goes into the csvlog, not argue that
something which is clearly useful be excluded.

There's already a patch out there for adding a way to control what goes
into csvlog and it probably wouldn't be too hard to update it.  I know,
because I wrote it.  I continue to feel that it would be useful to add
and that tools which need to deal with csvlogs should be updated to
handle being told what's in it, just like pgBadger does for our much
more complicated and painful regular log files.  That would remove these
bars to moving forward with more flexibility in log_line_prefix and
csvlog output.

Thanks!

Stephen

Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Robert Haas
Date:
On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote:
> I'm with Thomas on this and I disagree that the "csvlog bloat" argument
> has merit.  If we're worried about bloat in csv then we should provide a
> way for users to control what goes into the csvlog, not argue that
> something which is clearly useful be excluded.

I agree, but I think if we do that it would be highly desirable do
something to make the format discoverable.  For example, at the
beginning of each file and whenever the format changes, we write some
kind of unmistakable header line into the file identifying what fields
will be present on each line thereafter.  It's undesirable for log
analysis tools to need to contain logic that tries (imperfectly, no
doubt) to reverse-engineer the field list.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Alvaro Herrera
Date:
Robert Haas wrote:
> On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > I'm with Thomas on this and I disagree that the "csvlog bloat" argument
> > has merit.  If we're worried about bloat in csv then we should provide a
> > way for users to control what goes into the csvlog, not argue that
> > something which is clearly useful be excluded.
>
> I agree, but I think if we do that it would be highly desirable do
> something to make the format discoverable.  For example, at the
> beginning of each file and whenever the format changes, we write some
> kind of unmistakable header line into the file identifying what fields
> will be present on each line thereafter.  It's undesirable for log
> analysis tools to need to contain logic that tries (imperfectly, no
> doubt) to reverse-engineer the field list.

Hmm, I wouldn't use the same file for different formats.  If the format
is changed, a different file should be used.

--
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Re: [ADMIN] 9.5 new setting "cluster name" and logging

From
Tom Lane
Date:
Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> I'm with Thomas on this and I disagree that the "csvlog bloat" argument
>> has merit.  If we're worried about bloat in csv then we should provide a
>> way for users to control what goes into the csvlog, not argue that
>> something which is clearly useful be excluded.

> I agree, but I think if we do that it would be highly desirable do
> something to make the format discoverable.  For example, at the
> beginning of each file and whenever the format changes, we write some
> kind of unmistakable header line into the file identifying what fields
> will be present on each line thereafter.  It's undesirable for log
> analysis tools to need to contain logic that tries (imperfectly, no
> doubt) to reverse-engineer the field list.

Automatic "csv header" at the start of each logfile, perhaps?  And
force a logfile switch if it changes.

            regards, tom lane