Hi,
I ran into a smart/fast shutdown hang on master while looking at crash
restart. It happens in the window after we relaunch startup, but before
WAL redo starts. FatalError is still set, so we wait for the new
checkpointer and I/O workers after sending SIGTERM, which they ignore.
AbortStartTime has already been reset too, so nothing escalates to
SIGKILL.
I first tried routing that shutdown through
HandleFatalError(PMQUIT_FOR_STOP, false), i.e. sending SIGQUIT to the
children. That fixed the hang, but still went down the abnormal-shutdown
path and left the stale flag in place.
Then I remembered Noah's suggestion [1] in Justin's older thread on this
same hang [2]: clear FatalError when we relaunch startup. The attached v1
tries that instead, with a small TAP test. AFAICS, the old children are
gone and shmem is rebuilt by then. FWIW, if the new checkpointer crashes
in that window, we now do a full cleanup instead of silently respawning
it. Does this look like the right point to clear the flag?
One wrinkle is #19623 [3]. The special case from ead8f696b7c becomes
unreachable with this, so the patch removes it and puts back the
Assert(!FatalError) in HandleFatalError(). IIUC, if every new child
fails, we could now reap another child's exit before the startup
process's and go for another restart instead of giving up. Michael, do
you think that case still needs special handling? I haven't tested that
scenario.
I've added a WIP TAP test too which may need modifications.
(I haven't tried the patch on the back branches yet.)
Regards,
Ayush
[1]
https://www.postgresql.org/message-id/20241025181701.ed.nmisch@google.com[2]
https://www.postgresql.org/message-id/flat/ZWlrdQarrZvLsgIk@pryzbyj2023 (Thomas's patch there is CF 4884)
[3]
https://www.postgresql.org/message-id/flat/19623-f9bd331940be1273@postgresql.org
