Thread: Segmentation fault with PG-12
In our production-environment we get sig11 every now and then after upgrading to PG-12:
2019-10-08 15:45:29.654 CEST [8829-76] LOG: server process (PID 20631) was terminated by signal 11: Segmentation fault
2019-10-08 15:45:29.654 CEST [8829-77] DETAIL: Failed process was running: COMMIT
2019-10-08 15:45:29.654 CEST [8829-78] LOG: terminating any other active server processes
2019-10-08 15:45:29.654 CEST [8829-77] DETAIL: Failed process was running: COMMIT
2019-10-08 15:45:29.654 CEST [8829-78] LOG: terminating any other active server processes
Will running a debug-enabled build slow things noticably down?
Is there a way to make it dump a stack-trace (or back-trace in C-land?) on sig11?
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
På tirsdag 08. oktober 2019 kl. 17:24:21, skrev Tom Lane <tgl@sss.pgh.pa.us>:
Andreas Joseph Krogh <andreas@visena.com> writes:
> Will running a debug-enabled build slow things noticably down?
gcc promises that the generated code is the same with or without debug.
I think clang does too. With other compilers you may pay some penalty.
Nice, I'm using the ubuntu-packages, so I'll go ahead and install postgresql-12-dbgsym
> Is there a way
> to make it dump a stack-trace (or back-trace in C-land?) on sig11?
You should be able to get a core file from which you can extract a
stack trace (and other info) after the fact.
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
I'll look into that, thanks.
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
På tirsdag 08. oktober 2019 kl. 17:24:21, skrev Tom Lane <tgl@sss.pgh.pa.us>:
Andreas Joseph Krogh <andreas@visena.com> writes:
> Will running a debug-enabled build slow things noticably down?
gcc promises that the generated code is the same with or without debug.
I think clang does too. With other compilers you may pay some penalty.
> Is there a way
> to make it dump a stack-trace (or back-trace in C-land?) on sig11?
You should be able to get a core file from which you can extract a
stack trace (and other info) after the fact.
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
regards, tom lane
Attached is output from "bt full".
Is this helpful?
Anything else I can do to help narrowing down the problem?
Thanks.
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
På onsdag 09. oktober 2019 kl. 16:16:37, skrev Tom Lane <tgl@sss.pgh.pa.us>:
Andreas Joseph Krogh <andreas@visena.com> writes:
> Attached is output from "bt full". Is this helpful?
Well, it shows that the failure is occurring while trying to evaluate
a variable in a trigger's WHEN clause during
"UPDATE origo_email_delivery SET folder_id=$1, created=$2\nWHERE entity_id IN ($3)\nRETURNING entity_id"
And I'd bet that the root cause is something to do with Andres' tuple slot
work. But (at least to my eye) it's not apparent exactly what's wrong.
Can you show us the table definition and associated trigger definitions
for origo_email_delivery?
This doesn't seem to correlate with your original report, btw,
as that claimed the crash was during COMMIT.
regards, tom lane
FWIW: It doesn't always happen when that UPDATE-statement is issued, so it's not reproducable. We'll see what the next core-dump gives us.
Is it OK if I send you the table/trigger-definitions off-list?
-- 
Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
Attachment
Attachment
På torsdag 10. oktober 2019 kl. 07:25:26, skrev Andres Freund <andres@anarazel.de>:
On 2019-10-09 10:16:37 -0400, Tom Lane wrote:
> Andreas Joseph Krogh <andreas@visena.com> writes:
> > Attached is output from "bt full". Is this helpful?
>
> Well, it shows that the failure is occurring while trying to evaluate
> a variable in a trigger's WHEN clause during
> "UPDATE origo_email_delivery SET folder_id=$1, created=$2\nWHERE entity_id IN ($3)\nRETURNING entity_id"
> And I'd bet that the root cause is something to do with Andres' tuple slot
> work. But (at least to my eye) it's not apparent exactly what's wrong.
It looks like this could "just" be another report of #16036, which was
already fixed in:
commit d986d4e87f61c68f52c68ebc274960dc664b7b4e
Author: Andres Freund <andres@anarazel.de>
Date: 2019-10-04 11:59:34 -0700
Fix crash caused by EPQ happening with a before update trigger present.
(Tom: This mail is only viewable as text/html, to if you're reading the text/plain version it will seem "hashed")
Aha, that whould be 60e97d63e5d19098e11fa32431a20eea820e2ae9 in REL_12_STABLE
We'll build and run HEAD of REL_12_STABLE, and report back.
> This doesn't seem to correlate with your original report, btw,
> as that claimed the crash was during COMMIT.
That however, would be confusing, unless there's some deferred trigger
that causes another update, which then fires a before update trigger
causing the problem.
Greetings,
Andres Freund
We have a deferred trigger which updates origo_email_delivery:
CREATE OR REPLACE FUNCTION origo_index_email_props_tf() RETURNS TRIGGER AS $$ declare v_prop origo_email_message_property; BEGIN v_prop := NEW; UPDATE origo_email_delivery SET is_seen = v_prop.is_seen, followup_id = v_prop.followup_id, is_replied = v_prop.is_replied, is_forwarded = v_prop.is_forwarded, is_draft = v_prop.is_draft, is_done = v_prop.is_done, is_flagged = v_prop.is_flagged, modseq = greatest(modseq, v_prop.modseq) WHERE message_id = v_prop.message_id AND owner_id = v_prop.owner_id; RETURN NEW; END; $$ LANGUAGE plpgsql; CREATE CONSTRAINT TRIGGER origo_index_email_props_t AFTER INSERT OR UPDATE ON origo_email_message_property DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE PROCEDURE origo_index_email_props_tf();
.. and then trigger the following UPDATE-trigger:
CREATE TRIGGER origo_email_delivery_update_t BEFORE UPDATE ON origo_email_delivery FOR EACH ROW WHEN (OLD.folder_id <> NEW.folder_id OR NEW.is_deleted <> OLD.is_deleted) EXECUTE PROCEDURE origo_email_delivery_update_tf();
Maybe that will trigger the bug.
Thanks.
--
