Thread: Why has postmaster shutdown gotten so slow?
Shutdown of an idle postmaster used to take about two or three seconds
(mostly due to the sync/sleep(2)/sync in md_sync). For the last couple
of days it's taking more like a dozen seconds. I presume somebody broke
something, but I'm unsure whether to pin the blame on bgwriter or
Windows changes. Anyone care to fess up?
regards, tom lane
> Shutdown of an idle postmaster used to take about two or three seconds > (mostly due to the sync/sleep(2)/sync in md_sync). For the last couple > of days it's taking more like a dozen seconds. I presume somebody broke > something, but I'm unsure whether to pin the blame on bgwriter or > Windows changes. Anyone care to fess up? AFAICS, Win32 changes for the past few days have been minimal, and pretty much isolated to Win32. Happy to stand corrected, but I'd start by looking elsewhere... Cheers, Claudio --- Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see <a href="http://www.memetrics.com/emailpolicy.html">http://www.memetrics.com/em ailpolicy.html</a>
Tom Lane wrote: > Shutdown of an idle postmaster used to take about two or three seconds > (mostly due to the sync/sleep(2)/sync in md_sync). For the last couple > of days it's taking more like a dozen seconds. I presume somebody broke > something, but I'm unsure whether to pin the blame on bgwriter or > Windows changes. Anyone care to fess up? I guess it could well be the bgwriter, which when having nothing to do at all is sleeping for 10 seconds. Not sure, will check. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Jan Wieck wrote: > Tom Lane wrote: > >> Shutdown of an idle postmaster used to take about two or three seconds >> (mostly due to the sync/sleep(2)/sync in md_sync). For the last couple >> of days it's taking more like a dozen seconds. I presume somebody broke >> something, but I'm unsure whether to pin the blame on bgwriter or >> Windows changes. Anyone care to fess up? > > I guess it could well be the bgwriter, which when having nothing to do > at all is sleeping for 10 seconds. Not sure, will check. I checked the background writer for this and I can not reproduce the behaviour. If the bgwriter had zero blocks to write it does PG_USLEEP for 10 seconds, which on Unix is done by select() and that is correctly interrupted when the postmaster sends it the term signal on shutdown. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Jan Wieck <JanWieck@Yahoo.com> writes:
> I checked the background writer for this and I can not reproduce the
> behaviour. If the bgwriter had zero blocks to write it does PG_USLEEP
> for 10 seconds, which on Unix is done by select() and that is correctly
> interrupted when the postmaster sends it the term signal on shutdown.
This appears to be a platform-dependent behavior. The HPUX select(2) man
page says
[EINTR] The select() function was interrupted before any of the selected events
occurredand before the timeout interval expired. If SA_RESTART has been
setfor the interrupting signal, it is implementation-dependent whether select() restarts
or returns with EINTR.
which text also appears verbatim in the Single Unix Spec. Since we set
SA_RESTART for every signal except SIGALRM (see pqsignal.c), we are
subject to the implementation dependency for SIGTERM.
Tracing the bgwriter process on my machine makes it real obvious that in
fact the select delay is allowed to finish out when SIGTERM is received.
In fact worse than that: it's restarted from the beginning. If 5
seconds have already elapsed, another 10 still elapse before the select
exits.
This won't do :-(. We cannot afford to fritter away 10 seconds in the
SIGTERM shutdown cycle --- on typical systems init isn't going to give
us more than 20 seconds before a hard kill.
I'd suggest reducing the delay to a second or two, or perhaps breaking
it into several 1-second waits with interrupt flag checks between.
In the longer run we might want to rethink what we are doing with
SA_RESTART, but I am not sure about the implications of fooling with
that.
regards, tom lane
Tom Lane wrote: > Jan Wieck <JanWieck@Yahoo.com> writes: >> I checked the background writer for this and I can not reproduce the >> behaviour. If the bgwriter had zero blocks to write it does PG_USLEEP >> for 10 seconds, which on Unix is done by select() and that is correctly >> interrupted when the postmaster sends it the term signal on shutdown. > > This appears to be a platform-dependent behavior. The HPUX select(2) man > page says > > [EINTR] The select() function was interrupted before any > of the selected events occurred and before the > timeout interval expired. If SA_RESTART has been > set for the interrupting signal, it is > implementation-dependent whether select() restarts > or returns with EINTR. > > which text also appears verbatim in the Single Unix Spec. Since we set > SA_RESTART for every signal except SIGALRM (see pqsignal.c), we are > subject to the implementation dependency for SIGTERM. That explains it. > > Tracing the bgwriter process on my machine makes it real obvious that in > fact the select delay is allowed to finish out when SIGTERM is received. > In fact worse than that: it's restarted from the beginning. If 5 > seconds have already elapsed, another 10 still elapse before the select > exits. > > This won't do :-(. We cannot afford to fritter away 10 seconds in the > SIGTERM shutdown cycle --- on typical systems init isn't going to give > us more than 20 seconds before a hard kill. > > I'd suggest reducing the delay to a second or two, or perhaps breaking > it into several 1-second waits with interrupt flag checks between. > > In the longer run we might want to rethink what we are doing with > SA_RESTART, but I am not sure about the implications of fooling with > that. I think we should at this point have some maximum value for PG_xSLEEP over which it falls back to a function call that does either this breaking up into a loop with checking InterruptPending or removes the SA_RESTART flag while wating for the timeout. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
>> Tracing the bgwriter process on my machine makes it real >obvious that in >> fact the select delay is allowed to finish out when SIGTERM >is received. >> In fact worse than that: it's restarted from the beginning. If 5 >> seconds have already elapsed, another 10 still elapse before >the select >> exits. >> >> This won't do :-(. We cannot afford to fritter away 10 >seconds in the >> SIGTERM shutdown cycle --- on typical systems init isn't >going to give >> us more than 20 seconds before a hard kill. >> >> I'd suggest reducing the delay to a second or two, or >perhaps breaking >> it into several 1-second waits with interrupt flag checks between. >> >> In the longer run we might want to rethink what we are doing with >> SA_RESTART, but I am not sure about the implications of fooling with >> that. > >I think we should at this point have some maximum value for PG_xSLEEP >over which it falls back to a function call that does either this >breaking up into a loop with checking InterruptPending or removes the >SA_RESTART flag while wating for the timeout. If you look at my win32 signals patch nr 3 (posted feb 4th), I have code to do this for win32 in it. It breaks up select() timeouts into pieces of 1 second and polls for win32 signals inbetween. Turns out it wasn't necessary, since win32 *does* deliver our signals whlie in select. So for once it's win32 that does what we want - I think that's a first.. But it might help on another platform. //Magnus