On Fri, Dec 5, 2014 at 1:29 PM, Josh Berkus <josh@agliodbs.com> wrote:
>> We made some changes which decreased query cancel (optimizing queries,
>> turning on hot_standby_feedback) and we haven't seen a segfault since
>> then. As far as the user is concerned, this solves the problem, so I'm
>> never going to get a trace or a core dump file.
>
> Forgot a major piece of evidence as to why I think this is related to
> query cancel: in each case, the segfault was preceeded by a
> multi-backend query cancel 3ms to 30ms beforehand. It is possible that
> the backend running the query which segfaulted might have been the only
> backend *not* cancelled due to query conflict concurrently.
> Contradicting this, there are other multi-backend query cancels in the
> logs which do NOT produce a segfault.
I wonder if it would be useful to add additional instrumentation so
that even without a core dump, there was some cursory information
about the nature of a segfault.
Yes, doing something with a SIGSEGV handler is very scary, and there
are major portability concerns (e.g.
https://bugs.ruby-lang.org/issues/9654), but I believe it can be made
robust on Linux. For what it's worth, this open source project offers
that kind of functionality in the form of a library:
https://github.com/vmarkovtsev/DeathHandler
--
Peter Geoghegan
