Thread: pg_restore with Fc fails with [archiver] out of memory error

Created a pg_dump with Fc (custom format compression) that resulted in a
300+MB file. Now trying to pg_restore this thing fails with either an
out of memory error (as in the subject line) on FreeBSD 4.10-STABLE or
a:
pg_restore in malloc(): error: allocation failed
Abort (core dumped)
error on FreeBSD 5.2.1-P9 (RELEASE)

In both cases I have increased the max data segment size to 1.5GB and
the max stack size to 768M or so.

So obviously we have some error message when malloc'ing for the AR
structure. Any ideas here? Is it trying to stuff the 300MB file into
memory and uncompress and sort it there? Has anyone found a workaround
for this? (I have found similar questions and zero answers doing google
searches and mailing list searches). Is there any way of uncompressing
this file *other* than using pg_restore?

Sven

p.s. Although I will check the archives, I would appreciate a CC on any
responses.

Tom Lane wrote:
> Sven Willenberger <sven@dmv.com> writes:
>
>>pg_restore in malloc(): error: allocation failed
>>Abort (core dumped)
>
>
> A stack trace from that core file would be useful in figuring out what
> the problem is...  also, what PG version is this?
>
>             regards, tom lane

psql -V
psql (PostgreSQL) 7.4.3
contains support for command-line editing

gdb:
Reading symbols from /usr/local/lib/libpq.so.3...done.
Loaded symbols for /usr/local/lib/libpq.so.3
Reading symbols from /usr/local/lib/libintl.so.6...done.
Loaded symbols for /usr/local/lib/libintl.so.6
Reading symbols from /usr/local/lib/libssl.so.3...done.
Loaded symbols for /usr/local/lib/libssl.so.3
Reading symbols from /usr/local/lib/libcrypto.so.3...done.
Loaded symbols for /usr/local/lib/libcrypto.so.3
Reading symbols from /lib/libz.so.2...done.
Loaded symbols for /lib/libz.so.2
Reading symbols from /lib/libreadline.so.4...done.
Loaded symbols for /lib/libreadline.so.4
Reading symbols from /lib/libcrypt.so.2...done.
Loaded symbols for /lib/libcrypt.so.2
Reading symbols from /lib/libm.so.2...done.
Loaded symbols for /lib/libm.so.2
Reading symbols from /lib/libutil.so.4...done.
Loaded symbols for /lib/libutil.so.4
Reading symbols from /lib/libc.so.5...done.
Loaded symbols for /lib/libc.so.5
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x282a3dcf in kill () from /lib/libc.so.5
(gdb) bt
#0  0x282a3dcf in kill () from /lib/libc.so.5
#1  0x28298878 in raise () from /lib/libc.so.5
#2  0x28310f82 in abort () from /lib/libc.so.5
#3  0x2830f6fe in tcflow () from /lib/libc.so.5
#4  0x2830ff9b in tcflow () from /lib/libc.so.5
#5  0x28310356 in malloc () from /lib/libc.so.5
#6  0x0804c9a1 in ReadStr (AH=0x805e000) at pg_backup_archiver.c:1530
#7  0x0804d2d8 in ReadToc (AH=0x805e000) at pg_backup_archiver.c:1878
#8  0x0804fa94 in InitArchiveFmt_Custom (AH=0x805e000) at
pg_backup_custom.c:196
#9  0x0804cf7b in _allocAH (FileSpec=0xbfbfed19 "./callingdata",
fmt=archCustom, compression=0, mode=archModeWrite)
     at pg_backup_archiver.c:1743
#10 0x0804a628 in OpenArchive (FileSpec=0x0, fmt=archUnknown) at
pg_backup_archiver.c:103
#11 0x0804a22a in main (argc=5, argv=0xbfbfebe8) at pg_restore.c:333
#12 0x08049b52 in _start ()

Frame 6 - 11:
(gdb) up 6
#6  0x0804c9a1 in ReadStr (AH=0x805e000) at pg_backup_archiver.c:1530
1530    pg_backup_archiver.c: No such file or directory.
         in pg_backup_archiver.c
(gdb) up
#7  0x0804d2d8 in ReadToc (AH=0x805e000) at pg_backup_archiver.c:1878
1878    in pg_backup_archiver.c
(gdb) up
#8  0x0804fa94 in InitArchiveFmt_Custom (AH=0x805e000) at
pg_backup_custom.c:196
196     pg_backup_custom.c: No such file or directory.
         in pg_backup_custom.c
(gdb) up
#9  0x0804cf7b in _allocAH (FileSpec=0xbfbfed19 "./callingdata",
fmt=archCustom, compression=0, mode=archModeWrite)
     at pg_backup_archiver.c:1743
1743    pg_backup_archiver.c: No such file or directory.
         in pg_backup_archiver.c
(gdb) up
#10 0x0804a628 in OpenArchive (FileSpec=0x0, fmt=archUnknown) at
pg_backup_archiver.c:103
103     in pg_backup_archiver.c
(gdb) up
#11 0x0804a22a in main (argc=5, argv=0xbfbfebe8) at pg_restore.c:333
333     pg_restore.c: No such file or directory.
         in pg_restore.c

Let me know if you want any more details, a disassembled frame, what
have you.


Thanks,
Sven

Sven Willenberger <sven@dmv.com> writes:
> pg_restore in malloc(): error: allocation failed
> Abort (core dumped)

A stack trace from that core file would be useful in figuring out what
the problem is...  also, what PG version is this?

            regards, tom lane

On Tue, 2004-08-10 at 19:28 -0400, Sven Willenberger wrote:
>
> Tom Lane wrote:
> > Sven Willenberger <sven@dmv.com> writes:
> >
> >>pg_restore in malloc(): error: allocation failed
> >>Abort (core dumped)
> >
> >
> > A stack trace from that core file would be useful in figuring out what
> > the problem is...  also, what PG version is this?
> >
> >             regards, tom lane
>
> psql -V
> psql (PostgreSQL) 7.4.3
> contains support for command-line editing
>
<snip>

> Frame 6 - 11:
> (gdb) up 6
> #6  0x0804c9a1 in ReadStr (AH=0x805e000) at pg_backup_archiver.c:1530
> 1530    pg_backup_archiver.c: No such file or directory.
>          in pg_backup_archiver.c
> (gdb) up
> #7  0x0804d2d8 in ReadToc (AH=0x805e000) at pg_backup_archiver.c:1878
> 1878    in pg_backup_archiver.c
> (gdb) up
> #8  0x0804fa94 in InitArchiveFmt_Custom (AH=0x805e000) at
> pg_backup_custom.c:196
> 196     pg_backup_custom.c: No such file or directory.
>          in pg_backup_custom.c
> (gdb) up
> #9  0x0804cf7b in _allocAH (FileSpec=0xbfbfed19 "./callingdata",
> fmt=archCustom, compression=0, mode=archModeWrite)
>      at pg_backup_archiver.c:1743
> 1743    pg_backup_archiver.c: No such file or directory.
>          in pg_backup_archiver.c
> (gdb) up
> #10 0x0804a628 in OpenArchive (FileSpec=0x0, fmt=archUnknown) at
> pg_backup_archiver.c:103
> 103     in pg_backup_archiver.c
> (gdb) up
> #11 0x0804a22a in main (argc=5, argv=0xbfbfebe8) at pg_restore.c:333
> 333     pg_restore.c: No such file or directory.
>          in pg_restore.c
>
> Let me know if you want any more details, a disassembled frame, what
> have you.

For future searches,etc, the problem was a corrupted dump file due to
ftp issues when transferring the file. Thanks, Tom, for helping diagnose
this.

Sven