Thread: less log level for success dynamic background workers for 9.5
Hi
I am working on scheduler extension for 9.5. It use bgworkers intensively for any task. This is reason, why I need to decrease a log level - and I am thinking so parallel computing needs it due high number of created and finished workers.Regards
Pavel
Attachment
On 6/14/15 12:25 AM, Pavel Stehule wrote: > Hi > > I am working on scheduler extension for 9.5. It use bgworkers > intensively for any task. This is reason, why I need to decrease a log > level - and I am thinking so parallel computing needs it due high number > of created and finished workers. > > It should be fixed in 9.5 - because it is limiting factor for bgworkers > usage in 9.5. Anything ever happen with this? I agree that LOG is to high for reporting most (if not all) of these things. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Data in Trouble? Get it in Treble! http://BlueTreble.com
On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote: > On 6/14/15 12:25 AM, Pavel Stehule wrote: >> I am working on scheduler extension for 9.5. It use bgworkers >> intensively for any task. This is reason, why I need to decrease a log >> level - and I am thinking so parallel computing needs it due high number >> of created and finished workers. >> >> It should be fixed in 9.5 - because it is limiting factor for bgworkers >> usage in 9.5. > > > Anything ever happen with this? I agree that LOG is to high for reporting > most (if not all) of these things. I think we should consider having a flag for this behavior rather than changing the behavior across the board. But then again, maybe we should just change it. What do others think? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote: > On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote: >> Anything ever happen with this? I agree that LOG is to high for reporting >> most (if not all) of these things. > > I think we should consider having a flag for this behavior rather than > changing the behavior across the board. > But then again, maybe we should just change it. > > What do others think? A GUC just for that looks like an overkill to me, this log is useful when debugging. And one could always have its bgworker call elog by itself at startup and before leaving to provide more or less similar information. -- Michael
Michael Paquier <michael.paquier@gmail.com> writes:
> On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote:
>> On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote:
>>> Anything ever happen with this? I agree that LOG is to high for reporting
>>> most (if not all) of these things.
>> I think we should consider having a flag for this behavior rather than
>> changing the behavior across the board.
>> But then again, maybe we should just change it.
>>
>> What do others think?
> A GUC just for that looks like an overkill to me, this log is useful
> when debugging. And one could always have its bgworker call elog by
> itself at startup and before leaving to provide more or less similar
> information.
I agree that we don't need YAGUC here, particularly not one that applies
indiscriminately to all bgworkers. I'd vote for just decreasing the log
level. The current coding is appropriate for a facility that's basically
experimental; but as it moves towards being something that would be used
routinely in production, the argument for being noisy in the log gets
weaker and weaker.
regards, tom lane
On Mon, Jun 22, 2015 at 9:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Michael Paquier <michael.paquier@gmail.com> writes: >> On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote: >>> On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote: >>>> Anything ever happen with this? I agree that LOG is to high for reporting >>>> most (if not all) of these things. > >>> I think we should consider having a flag for this behavior rather than >>> changing the behavior across the board. >>> But then again, maybe we should just change it. >>> >>> What do others think? > >> A GUC just for that looks like an overkill to me, this log is useful >> when debugging. And one could always have its bgworker call elog by >> itself at startup and before leaving to provide more or less similar >> information. > > I agree that we don't need YAGUC here, particularly not one that applies > indiscriminately to all bgworkers. I'd vote for just decreasing the log > level. The current coding is appropriate for a facility that's basically > experimental; but as it moves towards being something that would be used > routinely in production, the argument for being noisy in the log gets > weaker and weaker. I was thinking of a background worker flag, not a GUC. BGWORKER_QUIET, or something like that. But I guess we ought to just change it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
2015-06-23 15:20 GMT+02:00 Robert Haas <robertmhaas@gmail.com>:
I was thinking of a background worker flag, not a GUC.On Mon, Jun 22, 2015 at 9:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Michael Paquier <michael.paquier@gmail.com> writes:
>> On Tue, Jun 23, 2015 at 10:07 AM, Robert Haas wrote:
>>> On Mon, Jun 22, 2015 at 8:19 PM, Jim Nasby wrote:
>>>> Anything ever happen with this? I agree that LOG is to high for reporting
>>>> most (if not all) of these things.
>
>>> I think we should consider having a flag for this behavior rather than
>>> changing the behavior across the board.
>>> But then again, maybe we should just change it.
>>>
>>> What do others think?
>
>> A GUC just for that looks like an overkill to me, this log is useful
>> when debugging. And one could always have its bgworker call elog by
>> itself at startup and before leaving to provide more or less similar
>> information.
>
> I agree that we don't need YAGUC here, particularly not one that applies
> indiscriminately to all bgworkers. I'd vote for just decreasing the log
> level. The current coding is appropriate for a facility that's basically
> experimental; but as it moves towards being something that would be used
> routinely in production, the argument for being noisy in the log gets
> weaker and weaker.
BGWORKER_QUIET, or something like that. But I guess we ought to just
change it.
I have not any problem with bg worker flag. The only question is, what should be by default.
Pavel
On Tue, Jun 23, 2015 at 10:53 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote: > 2015-06-23 15:20 GMT+02:00 Robert Haas <robertmhaas@gmail.com>: >> I was thinking of a background worker flag, not a GUC. >> BGWORKER_QUIET, or something like that. But I guess we ought to just >> change it. > > I have not any problem with bg worker flag. The only question is, what > should be by default. Well, if the flag is BGWORKER_QUIET, then the default behavior remains unchanged, but when that flag is used, the log level is reduced to DEBUG1. That has the advantage of not breaking backward compatibility. But I'm not sure whether anyone cares if we just break it, and it's certainly simpler without the flag. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas wrote: > On Tue, Jun 23, 2015 at 10:53 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote: > > 2015-06-23 15:20 GMT+02:00 Robert Haas <robertmhaas@gmail.com>: > >> I was thinking of a background worker flag, not a GUC. > >> BGWORKER_QUIET, or something like that. But I guess we ought to just > >> change it. > > > > I have not any problem with bg worker flag. The only question is, what > > should be by default. > > Well, if the flag is BGWORKER_QUIET, then the default behavior remains > unchanged, but when that flag is used, the log level is reduced to > DEBUG1. That has the advantage of not breaking backward > compatibility. But I'm not sure whether anyone cares if we just break > it, and it's certainly simpler without the flag. I vote we do it the other way around, that is have a flag BGWORKER_VERBOSE. This breaks backwards compatibility (I don't think there's too much value in that in this case), but it copes with the more common use case that you want to have the flag while the worker is being developed; and things that are already working don't need to change in order to get the natural behavior. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Robert Haas wrote:
>> Well, if the flag is BGWORKER_QUIET, then the default behavior remains
>> unchanged, but when that flag is used, the log level is reduced to
>> DEBUG1. That has the advantage of not breaking backward
>> compatibility. But I'm not sure whether anyone cares if we just break
>> it, and it's certainly simpler without the flag.
> I vote we do it the other way around, that is have a flag BGWORKER_VERBOSE.
> This breaks backwards compatibility (I don't think there's too much
> value in that in this case), but it copes with the more common use case
> that you want to have the flag while the worker is being developed; and
> things that are already working don't need to change in order to get the
> natural behavior.
I concur: if we're to have a flag at all, it should work as Alvaro says.
However, I'm not real sure we need a flag. I think the use-case of
wanting extra logging for a bgworker under development is unlikely to be
satisfied very well by just causing existing start/stop logging messages
to come out at higher priority. You're likely to be wanting to log other,
bgworker-specific, events, and so you'll probably end up writing a bunch
of your own elog calls anyway (which you'll eventually remove, #ifdef out,
or decrease the log levels of).
regards, tom lane
On Tue, Jun 23, 2015 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: >> Robert Haas wrote: >>> Well, if the flag is BGWORKER_QUIET, then the default behavior remains >>> unchanged, but when that flag is used, the log level is reduced to >>> DEBUG1. That has the advantage of not breaking backward >>> compatibility. But I'm not sure whether anyone cares if we just break >>> it, and it's certainly simpler without the flag. > >> I vote we do it the other way around, that is have a flag BGWORKER_VERBOSE. >> This breaks backwards compatibility (I don't think there's too much >> value in that in this case), but it copes with the more common use case >> that you want to have the flag while the worker is being developed; and >> things that are already working don't need to change in order to get the >> natural behavior. > > I concur: if we're to have a flag at all, it should work as Alvaro says. > > However, I'm not real sure we need a flag. I think the use-case of > wanting extra logging for a bgworker under development is unlikely to be > satisfied very well by just causing existing start/stop logging messages > to come out at higher priority. You're likely to be wanting to log other, > bgworker-specific, events, and so you'll probably end up writing a bunch > of your own elog calls anyway (which you'll eventually remove, #ifdef out, > or decrease the log levels of). Yeah. So let's just change it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas wrote: > On Tue, Jun 23, 2015 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > However, I'm not real sure we need a flag. I think the use-case of > > wanting extra logging for a bgworker under development is unlikely to be > > satisfied very well by just causing existing start/stop logging messages > > to come out at higher priority. You're likely to be wanting to log other, > > bgworker-specific, events, and so you'll probably end up writing a bunch > > of your own elog calls anyway (which you'll eventually remove, #ifdef out, > > or decrease the log levels of). > > Yeah. So let's just change it. +1 -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 6/23/15 12:21 PM, Tom Lane wrote: > I concur: if we're to have a flag at all, it should work as Alvaro says. > > However, I'm not real sure we need a flag. I think the use-case of > wanting extra logging for a bgworker under development is unlikely to be > satisfied very well by just causing existing start/stop logging messages > to come out at higher priority. You're likely to be wanting to log other, > bgworker-specific, events, and so you'll probably end up writing a bunch > of your own elog calls anyway (which you'll eventually remove, #ifdef out, > or decrease the log levels of). FWIW, I have this problem *constantly* with plpgsql. I put RAISE DEBUGs in, but once you have those in enough places SET client_min_messages=debug becomes next to useless because of the huge volume of spew. What I'd like is a way to add an identifier to ereport/RAISE so you could turn on individual reports. If we had that we'd just make these particular ereports DEBUG1 and not worry about it because you could easily turn them on when needed. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Data in Trouble? Get it in Treble! http://BlueTreble.com
On 24 June 2015 at 03:23, Jim Nasby <Jim.Nasby@bluetreble.com> wrote: > On 6/23/15 12:21 PM, Tom Lane wrote: >> >> I concur: if we're to have a flag at all, it should work as Alvaro says. >> >> However, I'm not real sure we need a flag. I think the use-case of >> wanting extra logging for a bgworker under development is unlikely to be >> satisfied very well by just causing existing start/stop logging messages >> to come out at higher priority. You're likely to be wanting to log other, >> bgworker-specific, events, and so you'll probably end up writing a bunch >> of your own elog calls anyway (which you'll eventually remove, #ifdef out, >> or decrease the log levels of). > > > FWIW, I have this problem *constantly* with plpgsql. I put RAISE DEBUGs in, > but once you have those in enough places SET client_min_messages=debug > becomes next to useless because of the huge volume of spew. > > What I'd like is a way to add an identifier to ereport/RAISE so you could > turn on individual reports. If we had that we'd just make these particular > ereports DEBUG1 and not worry about it because you could easily turn them on > when needed. So, log identifiers/classifiers, essentially. Message numbers have been discussed before with regards to core and rejected consistently. I don't think it's really come up in terms of PLs and user-defined functions. I've certainly had similar issues to you w.r.t. to debug messages from user-level code, and wanted to be able to enable one particular log line, all log output from a particular function, or all log output from a particular extension / all functions in a schema. I think it's a whole separate topicto Pavel's original proposal though, and really merits a separate thread. For Pavel's issue I'm all in favour of just changing the log message, I think LOG is way too high for something that's internal implementation detail. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 6/23/15 8:11 PM, Craig Ringer wrote: > I've certainly had similar issues to you w.r.t. to debug messages from > user-level code, and wanted to be able to enable one particular log > line, all log output from a particular function, or all log output > from a particular extension / all functions in a schema. I would also like the ability to potentially change the log level for all calls made out of a certain function/schema/etc. Basically, change it for everything below a certain entry in the call stack. > I think it's a whole separate topicto Pavel's original proposal > though, and really merits a separate thread. For Pavel's issue I'm all > in favour of just changing the log message, I think LOG is way too > high for something that's internal implementation detail. +1. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Data in Trouble? Get it in Treble! http://BlueTreble.com
On Wed, Jun 24, 2015 at 7:39 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote: >> I think it's a whole separate topicto Pavel's original proposal >> though, and really merits a separate thread. For Pavel's issue I'm all >> in favour of just changing the log message, I think LOG is way too >> high for something that's internal implementation detail. > > +1. OK, I have committed Pavel's patch. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
2015-06-26 17:28 GMT+02:00 Robert Haas <robertmhaas@gmail.com>:
On Wed, Jun 24, 2015 at 7:39 PM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
>> I think it's a whole separate topicto Pavel's original proposal
>> though, and really merits a separate thread. For Pavel's issue I'm all
>> in favour of just changing the log message, I think LOG is way too
>> high for something that's internal implementation detail.
>
> +1.
OK, I have committed Pavel's patch.
Thank you
Pavel