large INSERT leads to "invalid memory alloc" - Mailing list pgsql-general

From Stefan Froehlich
Subject large INSERT leads to "invalid memory alloc"
Date
Msg-id 20121125171904.GA4226@89-149-202-102.internetserviceteam.com
Whole thread Raw
Responses Re: large INSERT leads to "invalid memory alloc"  (Pavel Stehule <pavel.stehule@gmail.com>)
interpret bytea output as text / double encode()  (Stefan Froehlich <postgresql@froehlich.priv.at>)
List pgsql-general
While converting a mysql database into postgres, I stumbled over the
following problem:

| INSERT INTO a (id, fkid, displayorder, name, description, internal, mimetype, mimedata, filesize) VALUES (73,6,5,
E'Seefeld.rar',E'Seefeld',0, E'application/octet-stream',decode('5261...0700', 'hex'),311484587); 

As the value for "filesize" suggests, this is a very large BYTEA
(formerly: LONGBLOB) entry with about 300 MB. This is untypical, all
other fields are about a couple of MB and don't make any problems.
This very line leads to:

| sfroehli@host:~$ psql dbname < statement.sql
| Password:
| ERROR:  invalid memory alloc request size 1073741824

I have not found any configuration directive similar to mysqls
"max_allowed_packet" to increase the buffer size. And besides, I
don't understand, why postgres wants to allocate 1 GB to store
300 MB (which take 600 MB of ASCII text in the decode()-string).

Any idea how to put this into the target database?

Stefan


pgsql-general by date:

Previous
From: "Kevin Grittner"
Date:
Subject: Re: Query caching absent "query caching"
Next
From: Pavel Stehule
Date:
Subject: Re: large INSERT leads to "invalid memory alloc"