Thread: gharial segfaulting on REL_12_STABLE only
Hi, This is apparently an EDB-owned machine but I have no access to it currently (I could ask if necessary). For some reason it's been failing for a week, but only on REL_12_STABLE, with this in the log: 2019-08-20 04:31:48.886 MDT [13421:4] LOG: server process (PID 13871) was terminated by signal 11: unrecognized signal 2019-08-20 04:31:48.886 MDT [13421:5] DETAIL: Failed process was running: SET default_table_access_method = ''; Apparently HPUX's sys_siglist doesn't recognise that most popular of signals, 11, but by googling I see that it has its traditional meaning there. That's clearly in the create_am test: 019-08-20 04:31:22.404 MDT [13871:31] pg_regress/create_am HINT: Use DROP ... CASCADE to drop the dependent objects too. 2019-08-20 04:31:22.404 MDT [13871:32] pg_regress/create_am STATEMENT: DROP ACCESS METHOD gist2; 2019-08-20 04:31:22.405 MDT [13871:33] pg_regress/create_am LOG: statement: DROP ACCESS METHOD gist2 CASCADE; 2019-08-20 04:31:22.422 MDT [13871:34] pg_regress/create_am LOG: statement: SET default_table_access_method = ''; Perhaps it was really running the next statement. It's hard to see how cdc8d371e2, the only non-doc commit listed on the first failure, could have anything to do with that. -- Thomas Munro https://enterprisedb.com
Thomas Munro <thomas.munro@gmail.com> writes:
> This is apparently an EDB-owned machine but I have no access to it
> currently (I could ask if necessary). For some reason it's been
> failing for a week, but only on REL_12_STABLE, with this in the log:
Yeah, I've been puzzling over that to little avail.
> It's hard to see how cdc8d371e2, the only non-doc commit listed on the
> first failure, could have anything to do with that.
Exactly :-(. It seems completely reproducible since then, but how
could that have triggered a failure over here? And why only in this
branch? The identical patch went into HEAD.
> 2019-08-20 04:31:48.886 MDT [13421:4] LOG: server process (PID 13871)
> was terminated by signal 11: unrecognized signal
> 2019-08-20 04:31:48.886 MDT [13421:5] DETAIL: Failed process was
> running: SET default_table_access_method = '';
> Apparently HPUX's sys_siglist doesn't recognise that most popular of
> signals, 11, but by googling I see that it has its traditional meaning
> there.
HPUX hasn't *got* sys_siglist, nor strsignal() which is what we're
actually relying on these days (cf. pgstrsignal.c). I was puzzled
by that too to start with, though. I wonder if we shouldn't rearrange
pg_strsignal so that the message in the !HAVE_STRSIGNAL case is
something like "signal names not available on this platform" rather
than something that looks like we should've recognized it and didn't.
> 2019-08-20 04:31:22.422 MDT [13871:34] pg_regress/create_am LOG:
> statement: SET default_table_access_method = '';
> Perhaps it was really running the next statement.
Hard to see how, because this should have reported
ERROR: invalid value for parameter "default_table_access_method": ""
DETAIL: default_table_access_method cannot be empty.
but it didn't get that far. It seems like it must have died either
in the (utterly trivial) check that leads to the above-quoted
complaint, or somewhere in the ereport mechanism. Neither theory
seems very credible.
The seeming action-at-a-distance nature of the failure has me
speculating about compiler or linker bugs, but I dislike
jumping to that type of conclusion without hard evidence.
A stack trace would likely be really useful right about now.
regards, tom lane
On Tue, Aug 27, 2019 at 1:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > A stack trace would likely be really useful right about now. Yeah. Looking into how to get that. -- Thomas Munro https://enterprisedb.com
On Tue, Aug 27, 2019 at 2:09 PM Thomas Munro <thomas.munro@gmail.com> wrote: > On Tue, Aug 27, 2019 at 1:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > A stack trace would likely be really useful right about now. > > Yeah. Looking into how to get that. Erm. I heard the system was in a very unhappy state and couldn't be logged into. After it was rebooted, the problem appears to have gone away. That is quite unsatisfying. "anole" runs on the same host, and occasionally fails to launch any parallel workers, and it seems to be pretty unhappy too -- very long runtimes (minutes where my smaller machines take seconds). So the machine may be massively overloaded and swapping or something like that, something to be looked into, but that doesn't explain how we get to a segfault without an underlying hard to reach bug in our code... -- Thomas Munro https://enterprisedb.com