Thread: postgres segfaulting on pg_restore

On 05/04/11 11:18, Chris Curvey wrote:

> This is on Ubuntu 10.10 server, 64-bit on Intel.  I'm running Postgres
> 9.0.3, 64-bit from the ppa repository.
>
> What can I do to help the developers find the root cause of this?  I'm
> happy to run a debug version, or find a corefile, or whatever I can, but
> I might need some guidance, as I'm pretty new to postgres.

A backtrace would be a good start. On Ubuntu you will need to install
debuginfo packages first. For details, see:

http://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD

--
Craig Ringer

On 7/04/2011 4:02 AM, Chris Curvey wrote:

> Hmmm.  I'm on Ubuntu, and running 9.0.3, so I'm getting my packages from
> Martin Pitt, per the instructions at
> http://www.dctrwatson.com/2010/09/installing-postgresql-9-0-on-ubuntu-10-04/.
>   I don't seem to see any debug packages there.

Nor I. https://launchpad.net/~pitti/+archive/postgresql/+packages
doesn't seem to list any.

Martin: Are your packages stripped, or do they include minimal debug
info in the main packages? Do you put your debuginfo debs anywhere
accessible?

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/

On 04/07/2011 06:37 PM, Chris Curvey wrote:

> 2) install bison, flex and libreadline5-dev (sudo apt-get install
> PACKAGE).  I already had the gcc package installed

The easy way to do this on Debian/ubuntu, if you're building something
that packages exist for, is (eg):

sudo apt-get build-dep postgresql

This downloads and installs all the dependencies required to compile
postgresql.

> And voila!  Here is the backtrace:

Thankyou VERY much for taking the time to collect this information.

It appears to be crashing while building an index:

CREATE INDEX itransf ON transactions USING btree
(loccode, startdtact, starttmact);

I don't know PostgreSQL's innards well enough to know much more than
that, but others do and may well check this out.

Are you able to distribute your dataset - if not to the general public,
then to someone interested in identifying the fault?

Does the same dataset crash Pg when restored on another machine?

> #0  0x00000000006ce317 in GetMemoryChunkSpace (pointer=0x347cc70) at
> mcxt.c:264
> #1  0x00000000006d3d56 in writetup_index (state=0x26fc530,
> tapenum=<value optimized out>, stup=<value optimized out>) at
> tuplesort.c:2924
> #2  0x00000000006d2af7 in dumptuples (state=0x26fc530, alltuples=0
> '\000') at tuplesort.c:2068
> #3  0x00000000006d392f in puttuple_common (state=0x26fc530,
> tuple=0x7fff1e21d3b0) at tuplesort.c:1097
> #4  0x00000000006d3c4c in tuplesort_putindextuple (state=0x26fc530,
> tuple=<value optimized out>) at tuplesort.c:943
> #5  0x0000000000472cac in btbuildCallback (index=<value optimized out>,
> htup=0x26f4460, values=<value optimized out>, isnull=<value optimized
> out>, tupleIsAlive=1 '\001', state=0x7fff1e21d870) at nbtree.c:194
> #6  0x00000000004ab1ec in IndexBuildHeapScan (heapRelation=<value
> optimized out>, indexRelation=<value optimized out>, indexInfo=<value
> optimized out>, allow_sync=<value optimized out>, callback=<value
> optimized out>, callback_state=<value optimized out>) at index.c:1866
> #7  0x0000000000472b35 in btbuild (fcinfo=<value optimized out>) at
> nbtree.c:123
> #8  0x00000000006b8ba1 in OidFunctionCall3 (functionId=<value optimized
> out>, arg1=140128587519600, arg2=140128587659696, arg3=40470992) at
> fmgr.c:1610
> #9  0x00000000004ab804 in index_build (heapRelation=0x7f723aae9670,
> indexRelation=0x7f723ab0b9b0, indexInfo=0x26989d0, isprimary=0 '\000')
> at index.c:1427
> #10 0x00000000004ad43e in index_create (heapRelationId=<value optimized
> out>, indexRelationName=<value optimized out>, indexRelationId=<value
> optimized out>, indexInfo=0x26989d0, indexColNames=<value optimized
> out>, accessMethodObjectId=<value optimized out>, tableSpaceId=0,
> classObjectId=0x26f2e70, coloptions=0x26f2e90, reloptions=0, isprimary=0
> '\000', isconstraint=0 '\000', deferrable=0 '\000', initdeferred=0
> '\000', allow_system_table_mods=<value optimized out>, skip_build=0
> '\000', concurrent=0 '\000') at index.c:959
> #11 0x0000000000514ec2 in DefineIndex (heapRelation=<value optimized
> out>, indexRelationName=<value optimized out>, indexRelationId=<value
> optimized out>, accessMethodName=<value optimized out>,
> tableSpaceName=<value optimized out>, attributeList=0x2, predicate=0x0,
> options=0x0, exclusionOpNames=0x0, unique=0 '\000', primary=0 '\000',
> isconstraint=0 '\000', deferrable=<value optimized out>,
> initdeferred=<value optimized out>, is_alter_table=0 '\000',
> check_rights=1 '\001', skip_build=0 '\000', quiet=0 '\000',
> concurrent=<value optimized out>) at indexcmds.c:484
> #12 0x0000000000603b69 in standard_ProcessUtility (parsetree=0x2648880,
> queryString=0x2647be0 "CREATE INDEX itransf ON transactions USING btree
> (loccode, startdtact, starttmact);", params=0x0, isTopLevel=1 '\001',
> dest=0x2648c20, completionTag=0x7fff1e21e3c0 "") at utility.c:876
> #13 0x00000000006000a7 in PortalRunUtility (portal=0x25bf0d0,
> utilityStmt=0x2648880, isTopLevel=0 '\000', dest=0x2648c20,
> completionTag=0x7fff1e21e3c0 "") at pquery.c:1191
> #14 0x00000000006010ec in PortalRunMulti (portal=0x25bf0d0, isTopLevel=1
> '\001', dest=0x2648c20, altdest=0x2648c20, completionTag=0x7fff1e21e3c0
> "") at pquery.c:1296
> #15 0x0000000000601852 in PortalRun (portal=<value optimized out>,
> count=<value optimized out>, isTopLevel=112 'p', dest=<value optimized
> out>, altdest=<value optimized out>, completionTag=<value optimized
> out>) at pquery.c:822
> #16 0x00000000005fde0b in exec_simple_query (query_string=<value
> optimized out>) at postgres.c:1058
> #17 0x00000000005fee47 in PostgresMain (argc=<value optimized out>,
> argv=<value optimized out>, username=<value optimized out>) at
> postgres.c:3931
> #18 0x00000000005cc3b9 in BackendRun () at postmaster.c:3555
> #19 BackendStartup () at postmaster.c:3242
> #20 ServerLoop () at postmaster.c:1431
> #21 0x00000000005cea1c in PostmasterMain (argc=39596208, argv=0x259f8d0)
> at postmaster.c:1092
> #22 0x0000000000575be0 in main (argc=3, argv=0x259f8c0) at main.c:188
>
> so, do I leave this here, or do I send it to pgsql-bugs?
>
>
> --
> Ignoring that little voice in my head since 1966!

On 04/07/2011 08:49 PM, Chris Curvey wrote:

> Let me see if I can load it somewhere else.   And since the only other
> machine I have to try it on is a Windows box, I will be able to answer
> my own question of "are pg_dump dumps OS-portable?"

The answer there is "mostly". Contrib modules can cause issues and
locale names differ across OSes. I usually CREATE DATABASE then restore
into the empty database, because pg_restore may try to create a database
using a locale that doesn't exist on the target machine.

--
Craig Ringer

Chris Curvey <chris@chriscurvey.com> writes:
> And voila!  Here is the backtrace:

> #0  0x00000000006ce317 in GetMemoryChunkSpace (pointer=0x347cc70) at
> mcxt.c:264
> #1  0x00000000006d3d56 in writetup_index (state=0x26fc530, tapenum=<value
> optimized out>, stup=<value optimized out>) at tuplesort.c:2924
> #2  0x00000000006d2af7 in dumptuples (state=0x26fc530, alltuples=0 '\000')
> at tuplesort.c:2068
> #3  0x00000000006d392f in puttuple_common (state=0x26fc530,
> tuple=0x7fff1e21d3b0) at tuplesort.c:1097
> #4  0x00000000006d3c4c in tuplesort_putindextuple (state=0x26fc530,
> tuple=<value optimized out>) at tuplesort.c:943
> #5  0x0000000000472cac in btbuildCallback (index=<value optimized out>,
> htup=0x26f4460, values=<value optimized out>, isnull=<value optimized out>,
> tupleIsAlive=1 '\001', state=0x7fff1e21d870) at nbtree.c:194

That is damn peculiar.  The tuple handed to writetup_index would have
been copied just moments before in tuplesort_putindextuple, so there is
no way that GetMemoryChunkSpace ought to fail.  If you do the run
several times over, do you get the exact same stack trace every time?

            regards, tom lane

Chris Curvey <chris@chriscurvey.com> writes:
> and, FWIW, here's another trace, which is NEARLY the same as the first one I
> posted, with the difference being a slightly different line number at #3.  I
> will be quiet now and leave the brain trust to ponder.  Let me know if there
> is anything else I can get for you guys.

> #0  0x00000000006ce317 in GetMemoryChunkSpace (pointer=0x292a0e0) at
> mcxt.c:264
> #1  0x00000000006d3d56 in writetup_index (state=0x2281670, tapenum=<value
> optimized out>, stup=<value optimized out>) at tuplesort.c:2924

It occurred to me that a simple explanation for a core dump there would
be if something had scribbled past the end of the preceding palloc
chunk.  That would tend to clobber the "context" link of the palloc
chunk after it, which would send GetMemoryChunkSpace off into
never-never land following a trashed pointer.

That doesn't get us very much closer to a solution, but it does suggest
that you might learn something if you try this with an assert-enabled
build (configure --enable-cassert).  Also, are there any contrib modules
or third-party add-on modules or home-brew C functions that you're using?

            regards, tom lane

Chris Curvey <chris@chriscurvey.com> writes:
> in reverse order:  no third-party, no contrib, no home-brew C.    here is a
> stack trace from a fresh build of 9.1 beta 1 built with enable-cassert.

>  Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000744777 in AllocSetAlloc (context=0x13f6d08, size=16) at
> aset.c:639
> 639 Assert(chunk->size >= size);
> #0  0x0000000000744777 in AllocSetAlloc (context=0x13f6d08, size=16) at
> aset.c:639

Hmm, that's pretty interesting.  Possibly it's just another
manifestation of something scribbling past the end of its allocated
chunk, but my credence for that theory is way down if there's no add-on
code involved.  Does the postmaster log show any messages like
    WARNING: detected write past chunk end in ...
before the crash?

            regards, tom lane

Chris Curvey <chris@chriscurvey.com> writes:
>> Hmm, that's pretty interesting.  Possibly it's just another
>> manifestation of something scribbling past the end of its allocated
>> chunk, but my credence for that theory is way down if there's no add-on
>> code involved.  Does the postmaster log show any messages like
>> WARNING: detected write past chunk end in ...
>> before the crash?

> Nope.  Just a bunch of messages saying "checkpoints occurring too
> frequently", then "server process (PID 123456) was terminated by signal 11:
>  Segmentation fault"

Hmph.  Well, I'm out of ideas here.  I think you're going to need to put
some effort into producing a self-contained test case so that others can
debug this ...

            regards, tom lane