Thread: dump/restore problem

Hello,

I have a dump created with pg_dump ver 7.4.1 from a postgres 7.4 database,
with encoding SQL_ASCII.
I'm trying to import this dump in a 7.4.1 postgres database with encoding
UNICODE, but I'm getting some errors:

psql:dump.sql:1277: ERROR:  invalid memory alloc request size
1073741824
psql:dump.sql:3490: ERROR:  invalid byte sequence for encoding
"UNICODE": 0xce20
CONTEXT:  COPY products, line 1, column ldesc: "<li>capacitate de spalare
5 kg^M
<li>viteza de centrifugare 800 rpm^M
<li>18 programe variate de spala..."
psql:dump.sql:5414: ERROR:  invalid memory alloc request size
1073741824
psql:dump.sql:5577: ERROR:  invalid byte sequence for encoding
"UNICODE": 0xd650
CONTEXT:  COPY feedback, line 26, column comments: "Stimata Doamna/Domn,^M
^M
As dori sa achizitionez un aparat de tuns Philishave HQC888, pe care, cu
toa..."


What can I do ?

Thanks.

Andrei Ivanov <andrei.ivanov@ines.ro> writes:
> I have a dump created with pg_dump ver 7.4.1 from a postgres 7.4 database,
> with encoding SQL_ASCII.
> I'm trying to import this dump in a 7.4.1 postgres database with encoding
> UNICODE, but I'm getting some errors:

> psql:dump.sql:1277: ERROR:  invalid memory alloc request size
> 1073741824
> psql:dump.sql:3490: ERROR:  invalid byte sequence for encoding
> "UNICODE": 0xce20

The "invalid byte sequence" problem occurs because you are trying to
load data that isn't valid UNICODE.  You need to fix the data.

I'm more interested in the "invalid memory alloc request" messages,
myself.  Could you show us the sections of the dump file that trigger
those errors?

            regards, tom lane

You will find more info attached.

What about the invalid UNICODE data ? How can I fix it ? In the attachment
there is the line that gives the error...

Thanks.

On Tue, 13 Jan 2004, Tom Lane wrote:

> Andrei Ivanov <andrei.ivanov@ines.ro> writes:
> > I have a dump created with pg_dump ver 7.4.1 from a postgres 7.4 database,
> > with encoding SQL_ASCII.
> > I'm trying to import this dump in a 7.4.1 postgres database with encoding
> > UNICODE, but I'm getting some errors:
>
> > psql:dump.sql:1277: ERROR:  invalid memory alloc request size
> > 1073741824
> > psql:dump.sql:3490: ERROR:  invalid byte sequence for encoding
> > "UNICODE": 0xce20
>
> The "invalid byte sequence" problem occurs because you are trying to
> load data that isn't valid UNICODE.  You need to fix the data.
>
> I'm more interested in the "invalid memory alloc request" messages,
> myself.  Could you show us the sections of the dump file that trigger
> those errors?
>
>             regards, tom lane
>

Andrei Ivanov <andrei.ivanov@ines.ro> writes:
> You will find more info attached.

I could not reproduce the "out of memory" failure using this information
(I got "invalid byte sequence" instead, on both of those COPY commands).
What exactly is the declaration of the tables being copied into?

> What about the invalid UNICODE data ? How can I fix it ?

It looks like your data is in a single-byte encoding, perhaps latin1 or
one of its siblings.  To convert to unicode you will first have to
identify just what encoding you're really using.  Then put a "set
client_encoding = whatever;" command at the top of the dump script (or
set it in postgresql.conf, if that seems easier) and you should be able
to import.

            regards, tom lane

I've set the encoding to LATIN2 and everything worked fine... no encoding
errors and no memory allocation failures...

The tables where the memory failures occured are:

CREATE TABLE press_releases (
    id serial NOT NULL,
    title character varying(255) NOT NULL,
    body text NOT NULL,
    tdate date DEFAULT ('NOW'::text)::date NOT NULL
);

CREATE TABLE symbols (
    id serial NOT NULL,
    name character varying(255) NOT NULL,
    description text NOT NULL
);

I was previously trying with encoding set to SQL_ASCII, because that was
the encoding of the database from which the data was taken from and where
everything was working fine with php, but I was having troubles with java
displaying the characters wrong... now I'm trying to see if with the data
converted to unicode works.

Thanks.

 On Wed, 14 Jan 2004, Tom Lane wrote:

> Andrei Ivanov <andrei.ivanov@ines.ro> writes:
> > You will find more info attached.
>
> I could not reproduce the "out of memory" failure using this information
> (I got "invalid byte sequence" instead, on both of those COPY commands).
> What exactly is the declaration of the tables being copied into?
>
> > What about the invalid UNICODE data ? How can I fix it ?
>
> It looks like your data is in a single-byte encoding, perhaps latin1 or
> one of its siblings.  To convert to unicode you will first have to
> identify just what encoding you're really using.  Then put a "set
> client_encoding = whatever;" command at the top of the dump script (or
> set it in postgresql.conf, if that seems easier) and you should be able
> to import.
>
>             regards, tom lane
>