Thread: statistics buffer is full
What does the message 'statistics buffer is full' mean? And if I saw lots of
these in my logs what sort of bad things would happen? Presumably this is
related to the planner statistics? Would I expect to have bad plans if this
happens and how would I resolve it? This is what I found in the source, but
I'm not really sure what's going on here:
Part of:
pgstat_recvbuffer(void)
<snip>
/*
* As long as we have buffer space we add the socket to the read
* descriptor set.
*/
if (msg_have <= (int) (PGSTAT_RECVBUFFERSZ - sizeof(PgStat_Msg))
)
{
FD_SET(pgStatSock, &rfds);
maxfd = pgStatSock;
overflow = false;
}
else
{
if (!overflow)
{
ereport(LOG,
(errmsg("statistics buffer is full")));
overflow = true;
}
}
--
Jeff Frost, Owner <jeff@frostconsultingllc.com>
Frost Consulting, LLC http://www.frostconsultingllc.com/
Phone: 650-780-7908 FAX: 650-649-1954
On Wed, 2006-11-08 at 18:46 -0800, Jeff Frost wrote:
> What does the message 'statistics buffer is full' mean? And if I saw lots of
> these in my logs what sort of bad things would happen? Presumably this is
> related to the planner statistics? Would I expect to have bad plans if this
> happens and how would I resolve it? This is what I found in the source, but
> I'm not really sure what's going on here:
Usually it means that you have stats_command_string turned on :)
>
> Part of:
>
> pgstat_recvbuffer(void)
> <snip>
> /*
> * As long as we have buffer space we add the socket to the read
> * descriptor set.
> */
> if (msg_have <= (int) (PGSTAT_RECVBUFFERSZ - sizeof(PgStat_Msg))
> )
> {
> FD_SET(pgStatSock, &rfds);
> maxfd = pgStatSock;
> overflow = false;
> }
> else
> {
> if (!overflow)
> {
> ereport(LOG,
> (errmsg("statistics buffer is full")));
> overflow = true;
> }
> }
>
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Wed, 8 Nov 2006, Joshua D. Drake wrote: > On Wed, 2006-11-08 at 18:46 -0800, Jeff Frost wrote: >> What does the message 'statistics buffer is full' mean? And if I saw lots of >> these in my logs what sort of bad things would happen? Presumably this is >> related to the planner statistics? Would I expect to have bad plans if this >> happens and how would I resolve it? This is what I found in the source, but >> I'm not really sure what's going on here: > > Usually it means that you have stats_command_string turned on :) hah! Does that mean it's just the buffering for pg_stat_activity and not for planner statistics? That doesn't make me feel too bad in this case. -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954
On Wed, 2006-11-08 at 19:29 -0800, Jeff Frost wrote:
> On Wed, 8 Nov 2006, Joshua D. Drake wrote:
>
> > On Wed, 2006-11-08 at 18:46 -0800, Jeff Frost wrote:
> >> What does the message 'statistics buffer is full' mean? And if I saw lots of
> >> these in my logs what sort of bad things would happen? Presumably this is
> >> related to the planner statistics? Would I expect to have bad plans if this
> >> happens and how would I resolve it? This is what I found in the source, but
> >> I'm not really sure what's going on here:
> >
> > Usually it means that you have stats_command_string turned on :)
>
> hah! Does that mean it's just the buffering for pg_stat_activity and not for
> planner statistics? That doesn't make me feel too bad in this case.
Yeah it is nothing to worry about, however turning it off is certainly a
performance benefit.
Joshua D. Drake
>
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Wed, 8 Nov 2006, Joshua D. Drake wrote: >> hah! Does that mean it's just the buffering for pg_stat_activity and not for >> planner statistics? That doesn't make me feel too bad in this case. > > Yeah it is nothing to worry about, however turning it off is certainly a > performance benefit. > So, then the next question: is that error because the command string was too long or because there were too many command strings in the buffer at once (i.e. it got backed up a bit and filled)? It would be a drag to be without pg_stat_activity on this system. The performance detriment is lessened in 8.2, correct? -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954
On Wed, 2006-11-08 at 21:46 -0800, Jeff Frost wrote:
> On Wed, 8 Nov 2006, Joshua D. Drake wrote:
>
> >> hah! Does that mean it's just the buffering for pg_stat_activity and not for
> >> planner statistics? That doesn't make me feel too bad in this case.
> >
> > Yeah it is nothing to worry about, however turning it off is certainly a
> > performance benefit.
> >
>
> So, then the next question: is that error because the command string was too
> long or because there were too many command strings in the buffer at once
> (i.e. it got backed up a bit and filled)?
This one I can't answer.
>
> It would be a drag to be without pg_stat_activity on this system. The
> performance detriment is lessened in 8.2, correct?
Yes my understanding is work was done on 8.2 to help eliviate the issue.
Sincerely,
Joshua D. Drake
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Wed, Nov 08, 2006 at 09:46:48PM -0800, Jeff Frost wrote: > On Wed, 8 Nov 2006, Joshua D. Drake wrote: > > >>hah! Does that mean it's just the buffering for pg_stat_activity and not > >>for > >>planner statistics? That doesn't make me feel too bad in this case. > > > >Yeah it is nothing to worry about, however turning it off is certainly a > >performance benefit. > > > > So, then the next question: is that error because the command string was > too long or because there were too many command strings in the buffer at > once (i.e. it got backed up a bit and filled)? IIRC, (in 8.1) the same amount of data is shuffled through the stats system for every command, so it'd be an issue of too many messages. > It would be a drag to be without pg_stat_activity on this system. The > performance detriment is lessened in 8.2, correct? Correct. Performance of updating the info for ps was also improved, iirc. -- Jim Nasby jim@nasby.net EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Jeff Frost wrote: > On Wed, 8 Nov 2006, Joshua D. Drake wrote: > > >>hah! Does that mean it's just the buffering for pg_stat_activity and not > >>for > >>planner statistics? That doesn't make me feel too bad in this case. > > > >Yeah it is nothing to worry about, however turning it off is certainly a > >performance benefit. > > > > So, then the next question: is that error because the command string was > too long or because there were too many command strings in the buffer at > once (i.e. it got backed up a bit and filled)? Yeah, it got backed up. It's an UDP socket and it's having trouble keeping up. Disabling stats_command_string removes the higher producer of "statistics" traffic, and the price you pay for it is that queries don't show up in pg_stat_activity. You can still see the tags in ps though, and of course you'll have them in the log if you enabled that. > It would be a drag to be without pg_stat_activity on this system. The > performance detriment is lessened in 8.2, correct? Yes, by not using the UDP socket to transmit the command string in the first place. They are now just stored in memory, which is considerably faster. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.