Thread: [HACKERS] pgbench stuck with 100% cpu usage
Hello,
--
While running some tests, I encountered a situation where pgbench gets stuck in an infinite loop, consuming 100% cpu. The setup was:
- Start postgres server from the master branch
- Initialise pgbench
- Run pgbench -c 10 -T 100
- Stop postgres with -m immediate
Now it seems that pgbench gets stuck and it's state machine does not advance. Attaching it to debugger, I saw that one of the clients remain stuck in this loop forever.
if (command->type == SQL_COMMAND)
{
if (!sendCommand(st, command))
{
/*
* Failed. Stay in CSTATE_START_COMMAND state, to
* retry. ??? What the point or retrying? Should
* rather abort?
*/
return;
}
else
st->state = CSTATE_WAIT_RESULT;
}
sendCommand() returns false because the underlying connection is bad and PQsendQuery returns 0. Reading the comment, it seems that the author thought about this situation but decided to retry instead of abort. Not sure what was the rationale for that decision, may be to deal with transient failures?
The commit that introduced this code is 12788ae49e1933f463bc. So I am copying Heikki.
Thanks,
Pavan
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
PostgreSQL Development, 24x7 Support, Training & Services
> While running some tests, I encountered a situation where pgbench gets
> stuck in an infinite loop, consuming 100% cpu. The setup was:
>
> - Start postgres server from the master branch
> - Initialise pgbench
> - Run pgbench -c 10 -T 100
> - Stop postgres with -m immediate
That is a strange test to run, but it would be better if the behavior was
not that one.
> Now it seems that pgbench gets stuck and it's state machine does not
> advance. Attaching it to debugger, I saw that one of the clients remain
> stuck in this loop forever.
>
> if (!sendCommand(st, command))
> {
> /*
> * Failed. Stay in CSTATE_START_COMMAND state, to
> * retry. ??? What the point or retrying? Should
> * rather abort?
> */
As the comments indicate and your situation shows, probably stopping the
client would be a better much option when send fails, instead of
retrying... indefinitely.
> The commit that introduced this code is 12788ae49e1933f463bc. So I amn
> copying Heikki.
AFAICR the commit was mostly a heavy restructuring of previous
unmaintainable spaghetti code. I'm not sure the problem was not there
before under one form or another.
I agree that it should error out & stop the client in this case at least.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
>> The commit that introduced this code is 12788ae49e1933f463bc. So I amn >> copying Heikki. > > AFAICR the commit was mostly a heavy restructuring of previous unmaintainable > spaghetti code. I'm not sure the problem was not there before under one form > or another. > > I agree that it should error out & stop the client in this case at least. Here is a probable "fix", which does was the comment said should be done. I could not trigger an infinite loop with various kill -9 and other quick stops. Could you try it on your side? -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Fri, Sep 29, 2017 at 12:22 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
While running some tests, I encountered a situation where pgbench gets
stuck in an infinite loop, consuming 100% cpu. The setup was:
- Start postgres server from the master branch
- Initialise pgbench
- Run pgbench -c 10 -T 100
- Stop postgres with -m immediate
That is a strange test to run, but it would be better if the behavior was not that one.
Thanks,
Pavan
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Sep 29, 2017 at 1:03 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
The commit that introduced this code is 12788ae49e1933f463bc. So I amn
copying Heikki.
AFAICR the commit was mostly a heavy restructuring of previous unmaintainable spaghetti code. I'm not sure the problem was not there before under one form or another.
I agree that it should error out & stop the client in this case at least.
Here is a probable "fix", which does was the comment said should be done.
Looks good to me.
I could not trigger an infinite loop with various kill -9 and other quick stops. Could you try it on your side?
Ok, I will try. But TBH I did not try to reproduce that either and I am not sure if I can. I discovered the problem when my laptop's battery started draining out much more quickly. Having seen the problem, it seems very obvious though.
Thanks,
Pavan
Pavan Deolasee http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
PostgreSQL Development, 24x7 Support, Training & Services
>>> - Run pgbench -c 10 -T 100 >>> - Stop postgres with -m immediate >> >> That is a strange test to run, but it would be better if the behavior was >> not that one. > > > Well, I think it's a very legitimate test, not for testing performance, but > testing crash recovery and I use it very often. Ok, interesting. Now I understand your purpose. You may consider something like "BEGIN; UPDATE ...; \sleep 100 ms; COMMIT;" so that a crash is most likely to occur with plenty transactions in progress but without much load. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, Sep 29, 2017 at 1:39 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote: > Looks good to me. Committed and back-patched to v10. I have to say I'm kind of surprised that the comment removed by this patch got committed in the first place. It's got a ??? in it and isn't very grammatical either. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
> Committed and back-patched to v10. I have to say I'm kind of > surprised that the comment removed by this patch got committed in the > first place. It's got a ??? in it and isn't very grammatical either. ISTM that I reviewed the initial patch. AFAICR I agreed with the comment that whether it was appropriate to go on was unclear, but it did not strike me as obviously a bad idea at the time, so I let it pass. Now it does strike me as a bad idea (tm):-) My English is kind of fuzzy, so I tend not to comment too much on English unless I'm really sure that it is wrong. Note that there is another 100% cpu pgbench bug, see https://commitfest.postgresql.org/15/1292/, which seems more likely to occur in the wild that this one, but there has been no review of the fix I sent. -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Sep 28, 2017 at 10:36 PM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote: > Well, I think it's a very legitimate test, not for testing performance, but > testing crash recovery and I use it very often. This particular test was run > to catch another bug which will be reported separately. Yeah, I use pgbench for stuff like that all the time. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers