Thread: ERROR: invalid memory alloc request size 4294967293

ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
Hello,

Actually I use postgresql version  8.4.6-0ubuntu10.04 in bacula server
to save information about backups from bacula.
But, 2 days ago, the postgresql make a error when I run the command
pg_dump. This is the error:

====================
02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: SQL command failed
02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: Error message
from server: ERROR:  invalid memory alloc request size 4294967293
02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: The command
was: COPY public.file (fileid, fileindex, jobid, pathid, filenameid,
markid, lstat, md5) TO stdout;
02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dumpall: pg_dump
failed on database "bacula", exiting
====================


In postgresql's logs show the same message:

====================
2011-01-02 06:31:07 CLST ERROR:  table "delcandidates" does not exist
2011-01-02 06:31:07 CLST STATEMENT:  DROP TABLE DelCandidates
2011-01-02 06:32:15 CLST ERROR:  invalid memory alloc request size 4294967293
2011-01-02 06:32:15 CLST STATEMENT:  COPY public.file (fileid,
fileindex, jobid, pathid, filenameid, markid, lstat, md5) TO stdout;
2011-01-02 06:32:15 CLST LOG:  could not receive data from client:
Connection reset by peer
2011-01-02 06:32:15 CLST LOG:  unexpected EOF on client connection
====================

I was searching in internet for a long time, and I found many
message/case/errors similar, but none solution !! :(

Until this moment, I have tried:

- checked the RAM... Is OK, I run memtest for 12 times and none error.
- checked the disk... Is OK, i run fsck and badblock and none error
- backuped the data... I rename the /var/lib/postgresql to
/var/lib/postgresql-old and after I copy the contain of
/var/lib/postgresql-old to /var/lib/postgresql and no errors
- checked the logs... I don't view importants messages of errors (only
the message above).
- changed the server... I move the actual data to new server, and I
have the same problem (with this test, I discard hardware problem)

If I run this command in the console:
su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc -l

in the first time, I receive a total of 1417599 lines
if a run this command for second time o more, I receive only 114
but, if I restart the server and I run this command again, so we have
the 1417599 again.

=========
su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc
ERROR:  invalid memory alloc request size 4294967293
1417599 32604777 173355793

su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc
ERROR:  invalid memory alloc request size 4294967293
   114    2622   14192

su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc
ERROR:  invalid memory alloc request size 4294967293
   114    2622   14192

su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc
ERROR:  invalid memory alloc request size 4294967293
1417599 32604777 173355793

su - postgres -c "psql bacula -c 'COPY public.file (fileid, fileindex,
jobid, pathid, filenameid, markid, lstat, md5) TO stdout'" | wc
ERROR:  invalid memory alloc request size 4294967293
   114    2622   14192
=========


This server is in a datacenter tougher a others many servers and with a UPS..
the uptime of this server was ~ 60 days (yesterday I restarted it).
is a dedicate server to postgresql and bacula with 8GB of RAM (ECC and
redundant)
and I ran the dump of database every day... and only two day ago, I
receive this error.


So.. In base a this I write to you to ask if everyone have any idea
about this problem ??

thanks and attentive.

--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
On Tue, Jan 4, 2011 at 10:59 AM, Bob Lunney <bob_lunney@yahoo.com> wrote:
> Run ulimit -a and verify the max memory size allowed for the postgres account.(I assume you are running postmaster
underthe postgres account, right?)  The allowed size should be large enough for the postmaster plus shared buffers and
severalother GUCs that require memory. 


Hello,

===============================
$ whoami
postgres

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
===============================

attentive.

--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Bob Lunney
Date:
Run ulimit -a and verify the max memory size allowed for the postgres account.(I assume you are running postmaster
underthe postgres account, right?)  The allowed size should be large enough for the postmaster plus shared buffers and
severalother GUCs that require memory.   

Bob Lunney


--- On Tue, 1/4/11, Victor Hugo dos Santos <listas.vhs@gmail.com> wrote:

> From: Victor Hugo dos Santos <listas.vhs@gmail.com>
> Subject: [ADMIN] ERROR: invalid memory alloc request size 4294967293
> To: pgsql-admin@postgresql.org
> Date: Tuesday, January 4, 2011, 8:48 AM
> Hello,
>
> Actually I use postgresql version  8.4.6-0ubuntu10.04
> in bacula server
> to save information about backups from bacula.
> But, 2 days ago, the postgresql make a error when I run the
> command
> pg_dump. This is the error:
>
> ====================
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> SQL command failed
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> Error message
> from server: ERROR:  invalid memory alloc request size
> 4294967293
> 02-Jan 06:32 bacula-dir JobId 31005:%2




Re: ERROR: invalid memory alloc request size 4294967293

From
Bob Lunney
Date:
Run ulimit -a and verify the max memory size allowed for the postgres account.(I assume you are running postmaster
underthe postgres account, right?)  The allowed size should be large enough for the postmaster plus shared buffers and
severalother GUCs that require memory.   

Bob Lunney


--- On Tue, 1/4/11, Victor Hugo dos Santos <listas.vhs@gmail.com> wrote:

> From: Victor Hugo dos Santos <listas.vhs@gmail.com>
> Subject: [ADMIN] ERROR: invalid memory alloc request size 4294967293
> To: pgsql-admin@postgresql.org
> Date: Tuesday, January 4, 2011, 8:48 AM
> Hello,
>
> Actually I use postgresql version  8.4.6-0ubuntu10.04
> in bacula server
> to save information about backups from bacula.
> But, 2 days ago, the postgresql make a error when I run the
> command
> pg_dump. This is the error:
>
> ====================
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> SQL command failed
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> Error message
> from server: ERROR:  invalid memory alloc request size
> 4294967293
> 02-Jan 06:32 bacula-dir JobId 31005:%2




Re: ERROR: invalid memory alloc request size 4294967293

From
Bob Lunney
Date:
Run ulimit -a and verify the max memory size allowed for the postgres account.(I assume you are running postmaster
underthe postgres account, right?)  The allowed size should be large enough for the postmaster plus shared buffers and
severalother GUCs that require memory.   

Bob Lunney


--- On Tue, 1/4/11, Victor Hugo dos Santos <listas.vhs@gmail.com> wrote:

> From: Victor Hugo dos Santos <listas.vhs@gmail.com>
> Subject: [ADMIN] ERROR: invalid memory alloc request size 4294967293
> To: pgsql-admin@postgresql.org
> Date: Tuesday, January 4, 2011, 8:48 AM
> Hello,
>
> Actually I use postgresql version  8.4.6-0ubuntu10.04
> in bacula server
> to save information about backups from bacula.
> But, 2 days ago, the postgresql make a error when I run the
> command
> pg_dump. This is the error:
>
> ====================
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> SQL command failed
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump:
> Error message
> from server: ERROR:  invalid memory alloc request size
> 4294967293
> 02-Jan 06:32 bacula-dir JobId 31005:%2




Re: ERROR: invalid memory alloc request size 4294967293

From
Thomas Markus
Date:
hi,

looks like a 32bit version so 4294967293 is too much (4GB). Use a 64bit
version

regards
Thomas


Am 04.01.2011 14:48, schrieb Victor Hugo dos Santos:
> Hello,
>
> Actually I use postgresql version  8.4.6-0ubuntu10.04 in bacula server
> to save information about backups from bacula.
> But, 2 days ago, the postgresql make a error when I run the command
> pg_dump. This is the error:
>
> ====================
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: SQL command failed
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: Error message
> from server: ERROR:  invalid memory alloc request size 4294967293
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dump: The command
> was: COPY public.file (fileid, fileindex, jobid, pathid, filenameid,
> markid, lstat, md5) TO stdout;
> 02-Jan 06:32 bacula-dir JobId 31005: BeforeJob: pg_dumpall: pg_dump
> failed on database "bacula", exiting
> ====================


Re: ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
On Tue, Jan 4, 2011 at 11:07 AM, Thomas Markus <t.markus@proventis.net> wrote:
> hi,
>
> looks like a 32bit version so 4294967293 is too much (4GB). Use a 64bit
> version

mmmm.. is a small table (look the size of last dump)

=============
$ ls -lh /tmp/public-files.sql
-rw-r----- 1 root root 298M Jan  4 11:59 /tmp/public-files.sql

$ sudo cat /tmp/public-files.sql  | wc
2580840 59359163 311685729
=============

but, I'm dowloading the 64bit image now and after I install it in a
Virtual Machine to try.

thanks and attentive to others commentaries.


--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
On Tue, Jan 4, 2011 at 12:02 PM, Victor Hugo dos Santos
<listas.vhs@gmail.com> wrote:

[...]

> but, I'm dowloading the 64bit image now and after I install it in a
> Virtual Machine to try.

I'm a stupid !! :(
I can't restore the same DB from 32bit in 64bits !!!
for this, should be export first in 32bits (pg_dump) and after import in 64bit.

but in my case, the pg_dump don't work !! :(

any idea ???

thanks

--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Tom Lane
Date:
Victor Hugo dos Santos <listas.vhs@gmail.com> writes:
> any idea ???

It looks like a corrupted-data problem from here.  You need to isolate
and delete the bad row(s).

            regards, tom lane

Re: ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
On Tue, Jan 4, 2011 at 1:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Victor Hugo dos Santos <listas.vhs@gmail.com> writes:
>> any idea ???
>
> It looks like a corrupted-data problem from here.  You need to isolate
> and delete the bad row(s).

hello

any sugestion about how found and delete the bad row(s) ??

thanks


--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Victor Hugo dos Santos
Date:
On Tue, Jan 4, 2011 at 1:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Victor Hugo dos Santos <listas.vhs@gmail.com> writes:
>> any idea ???
>
> It looks like a corrupted-data problem from here.  You need to isolate
> and delete the bad row(s).

Hello again...

this is very, very strange !! :D
(sorry for long message, but I'm trying to sent all information)


I was trying of isolate the "bad rows" in base a this article
http://archives.postgresql.org/pgsql-admin/2003-06/msg00204.php

first I create a function run a select the column "md5" (md5 and lstat
are both with errors) row-to-row..
after a time, I discoverer that the (first) row with problem is the "1417610"

So, now I run the command manually:

=========================
# su - postgres -c "psql bacula -c 'SELECT md5 from public.file OFFSET
1417610 LIMIT 1'"
             md5
-----------------------------
 WPC7vlHBLbGDA5XL6bsuBVsVVEM
(1 row)


# su - postgres -c "psql bacula -c 'SELECT md5 from public.file OFFSET
1417610 LIMIT 1'"
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
connection to server was lost


# su - postgres -c "psql bacula -c 'SELECT md5 from public.file OFFSET
1417610 LIMIT 1'"
             md5
-----------------------------
 WPC7vlHBLbGDA5XL6bsuBVsVVEM
(1 row)

# su - postgres -c "psql bacula -c 'SELECT md5 from public.file OFFSET
1417610 LIMIT 1'"
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
connection to server was lost
====================

In others words.. if I run the same command two times, I get a error,
and if I rerun the same command, work !!!
In the logs of postgresql I found after:


=========================
2011-01-04 16:40:35 CLST STATEMENT:  SELECT md5 from public.file
OFFSET 800000 LIMIT 1
2011-01-04 16:40:36 CLST DEBUG:  reaping dead processes
2011-01-04 16:40:36 CLST DEBUG:  server process (PID 15542) was
terminated by signal 11: Segmentation fault
2011-01-04 16:40:36 CLST LOG:  server process (PID 15542) was
terminated by signal 11: Segmentation fault
2011-01-04 16:40:36 CLST LOG:  terminating any other active server processes
2011-01-04 16:40:36 CLST DEBUG:  sending SIGQUIT to process 15533
2011-01-04 16:40:36 CLST DEBUG:  sending SIGQUIT to process 15534
2011-01-04 16:40:36 CLST DEBUG:  sending SIGQUIT to process 15535
2011-01-04 16:40:36 CLST DEBUG:  sending SIGQUIT to process 15536
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  reaping dead processes
2011-01-04 16:40:36 CLST LOG:  all server processes terminated; reinitializing
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(1): 3 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  invoking IpcMemoryCreate(size=32595968)
2011-01-04 16:40:36 CLST LOG:  database system was interrupted; last
known up at 2011-01-04 16:40:28 CLST
2011-01-04 16:40:36 CLST DEBUG:  checkpoint record is at 6/83C484E4
2011-01-04 16:40:36 CLST DEBUG:  redo record is at 6/83C484E4; shutdown TRUE
2011-01-04 16:40:36 CLST DEBUG:  next transaction ID: 0/416024; next OID: 156073
2011-01-04 16:40:36 CLST DEBUG:  next MultiXactId: 1; next MultiXactOffset: 0
2011-01-04 16:40:36 CLST LOG:  database system was not properly shut
down; automatic recovery in progress
2011-01-04 16:40:36 CLST LOG:  record with zero length at 6/83C48528
2011-01-04 16:40:36 CLST LOG:  redo is not required
2011-01-04 16:40:36 CLST DEBUG:  transaction ID wrap limit is
2147484295, limited by database "template1"
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(0): 3 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(0): 2 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  exit(0)
2011-01-04 16:40:36 CLST DEBUG:  shmem_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  proc_exit(-1): 0 callbacks to make
2011-01-04 16:40:36 CLST DEBUG:  reaping dead processes
2011-01-04 16:40:36 CLST LOG:  database system is ready to accept connections
2011-01-04 16:40:36 CLST LOG:  autovacuum launcher started
=========================

maybe the problem is in "6/83C48528" but none idea how fix it.

thanks.

--
--
Victor Hugo dos Santos
Linux Counter #224399

Re: ERROR: invalid memory alloc request size 4294967293

From
Tom Lane
Date:
Victor Hugo dos Santos <listas.vhs@gmail.com> writes:
> On Tue, Jan 4, 2011 at 1:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It looks like a corrupted-data problem from here. �You need to isolate
>> and delete the bad row(s).

> # su - postgres -c "psql bacula -c 'SELECT md5 from public.file OFFSET
> 1417610 LIMIT 1'"
> server closed the connection unexpectedly

So you've got some rows that are corrupted badly enough to crash the
backend :-(.

> In others words.. if I run the same command two times, I get a error,
> and if I rerun the same command, work !!!

When you're running long seqscans like these, you need to turn off
"synchronize_seqscans" to get reproducible results.  With that flag
turned on, scans may start from somewhere in the middle of the table
instead of always starting from the beginning.

            regards, tom lane