Thread: Re: [ADMIN] 9.5 new setting "cluster name" and logging
Hi, (x-posting to -hackers, more relevant audience) On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: > Now that there is a setting to give a cluster a "name", it would be nice to > have an escape sequence in the log_line_prefix setting that could reference > the cluster_name. I've argued[1][2] for this when cluster_name was introduced, but back then I seemed to have been the only one arguing for it. Josh later jumped that train. Given that we now had a number of people wishing for this, can we maybe reconsider? Greetings, Andres Freund [1] http://archives.postgresql.org/message-id/CADLWmXUm%3D7Y3UORZnGMdSC6p1eymO0k0JkH4NKr4KstdWk0P7g%40mail.gmail.com [2] http://archives.postgresql.org/message-id/20140505101007.GU12715@awork2.anarazel.de
On 02/08/2016 06:24 AM, Andres Freund wrote: > On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: >> Now that there is a setting to give a cluster a "name", it would be nice to >> have an escape sequence in the log_line_prefix setting that could reference >> the cluster_name. > > I've argued[1][2] for this when cluster_name was introduced, but back > then I seemed to have been the only one arguing for it. Josh later > jumped that train. > > Given that we now had a number of people wishing for this, can we maybe > reconsider? Seems like a reasonable idea to me. But if we add a log_line_prefix setting, shouldn't we also add it to csvlog output too? Joe -- Crunchy Data - http://crunchydata.com PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development
Attachment
On Tue, Feb 9, 2016 at 5:30 AM, Joe Conway <mail@joeconway.com> wrote: > On 02/08/2016 06:24 AM, Andres Freund wrote: >> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: >>> Now that there is a setting to give a cluster a "name", it would be nice to >>> have an escape sequence in the log_line_prefix setting that could reference >>> the cluster_name. >> >> I've argued[1][2] for this when cluster_name was introduced, but back >> then I seemed to have been the only one arguing for it. Josh later >> jumped that train. >> >> Given that we now had a number of people wishing for this, can we maybe >> reconsider? > > Seems like a reasonable idea to me. But if we add a log_line_prefix > setting, shouldn't we also add it to csvlog output too? Here's a tiny patch adding support for %C to log_line_prefix (this was part of the cluster_name patch that didn't go it). Given that csvlog's output format is hardcoded in write_csvlog, how is it supposed to evolve without upsetting consumers of this data? Wouldn't we first need to add a GUC that lets you control the columns it outputs? -- Thomas Munro http://www.enterprisedb.com
Attachment
Thomas, * Thomas Munro (thomas.munro@enterprisedb.com) wrote: > On Tue, Feb 9, 2016 at 5:30 AM, Joe Conway <mail@joeconway.com> wrote: > > On 02/08/2016 06:24 AM, Andres Freund wrote: > >> On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: > >>> Now that there is a setting to give a cluster a "name", it would be nice to > >>> have an escape sequence in the log_line_prefix setting that could reference > >>> the cluster_name. > >> > >> I've argued[1][2] for this when cluster_name was introduced, but back > >> then I seemed to have been the only one arguing for it. Josh later > >> jumped that train. > >> > >> Given that we now had a number of people wishing for this, can we maybe > >> reconsider? > > > > Seems like a reasonable idea to me. But if we add a log_line_prefix > > setting, shouldn't we also add it to csvlog output too? > > Here's a tiny patch adding support for %C to log_line_prefix (this was > part of the cluster_name patch that didn't go it). > > Given that csvlog's output format is hardcoded in write_csvlog, how is > it supposed to evolve without upsetting consumers of this data? > Wouldn't we first need to add a GUC that lets you control the columns > it outputs? Not sure if you really want to go there, but I do agree with you and there's a thread from a few years back about something similar: http://www.postgresql.org/message-id/flat/20110112142345.GA4933@tamriel.snowman.net Included in that thread is a patch, which likely requires some dusting off, to add exactly that ability. Thanks! Stephen
On Tue, Feb 9, 2016 at 3:24 AM, Andres Freund <andres@anarazel.de> wrote: > Hi, > > (x-posting to -hackers, more relevant audience) > > On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: >> Now that there is a setting to give a cluster a "name", it would be nice to >> have an escape sequence in the log_line_prefix setting that could reference >> the cluster_name. > > I've argued[1][2] for this when cluster_name was introduced, but back > then I seemed to have been the only one arguing for it. Josh later > jumped that train. > > Given that we now had a number of people wishing for this, can we maybe > reconsider? +1 I missed the part of this conversation that took place on -admin. I agree with Evan Rempel's post over there[1] that it's useful for syslog users, and that it's not ideal to have to hijack syslog_ident or explicitly copy the name into log_line_prefix. [1] https://www.postgresql.org/message-id/56C64017.7050303%40uvic.ca -- Thomas Munro http://www.enterprisedb.com
* Thomas Munro (thomas.munro@enterprisedb.com) wrote: > On Tue, Feb 9, 2016 at 3:24 AM, Andres Freund <andres@anarazel.de> wrote: > > (x-posting to -hackers, more relevant audience) > > > > On 2016-01-29 22:19:45 -0800, Evan Rempel wrote: > >> Now that there is a setting to give a cluster a "name", it would be nice to > >> have an escape sequence in the log_line_prefix setting that could reference > >> the cluster_name. > > > > I've argued[1][2] for this when cluster_name was introduced, but back > > then I seemed to have been the only one arguing for it. Josh later > > jumped that train. > > > > Given that we now had a number of people wishing for this, can we maybe > > reconsider? > > +1 > > I missed the part of this conversation that took place on -admin. I > agree with Evan Rempel's post over there[1] that it's useful for > syslog users, and that it's not ideal to have to hijack syslog_ident > or explicitly copy the name into log_line_prefix. > > [1] https://www.postgresql.org/message-id/56C64017.7050303%40uvic.ca I'm with Thomas on this and I disagree that the "csvlog bloat" argument has merit. If we're worried about bloat in csv then we should provide a way for users to control what goes into the csvlog, not argue that something which is clearly useful be excluded. There's already a patch out there for adding a way to control what goes into csvlog and it probably wouldn't be too hard to update it. I know, because I wrote it. I continue to feel that it would be useful to add and that tools which need to deal with csvlogs should be updated to handle being told what's in it, just like pgBadger does for our much more complicated and painful regular log files. That would remove these bars to moving forward with more flexibility in log_line_prefix and csvlog output. Thanks! Stephen
On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote: > I'm with Thomas on this and I disagree that the "csvlog bloat" argument > has merit. If we're worried about bloat in csv then we should provide a > way for users to control what goes into the csvlog, not argue that > something which is clearly useful be excluded. I agree, but I think if we do that it would be highly desirable do something to make the format discoverable. For example, at the beginning of each file and whenever the format changes, we write some kind of unmistakable header line into the file identifying what fields will be present on each line thereafter. It's undesirable for log analysis tools to need to contain logic that tries (imperfectly, no doubt) to reverse-engineer the field list. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas wrote: > On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote: > > I'm with Thomas on this and I disagree that the "csvlog bloat" argument > > has merit. If we're worried about bloat in csv then we should provide a > > way for users to control what goes into the csvlog, not argue that > > something which is clearly useful be excluded. > > I agree, but I think if we do that it would be highly desirable do > something to make the format discoverable. For example, at the > beginning of each file and whenever the format changes, we write some > kind of unmistakable header line into the file identifying what fields > will be present on each line thereafter. It's undesirable for log > analysis tools to need to contain logic that tries (imperfectly, no > doubt) to reverse-engineer the field list. Hmm, I wouldn't use the same file for different formats. If the format is changed, a different file should be used. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Oct 17, 2016 at 6:24 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> I'm with Thomas on this and I disagree that the "csvlog bloat" argument
>> has merit. If we're worried about bloat in csv then we should provide a
>> way for users to control what goes into the csvlog, not argue that
>> something which is clearly useful be excluded.
> I agree, but I think if we do that it would be highly desirable do
> something to make the format discoverable. For example, at the
> beginning of each file and whenever the format changes, we write some
> kind of unmistakable header line into the file identifying what fields
> will be present on each line thereafter. It's undesirable for log
> analysis tools to need to contain logic that tries (imperfectly, no
> doubt) to reverse-engineer the field list.
Automatic "csv header" at the start of each logfile, perhaps? And
force a logfile switch if it changes.
regards, tom lane