Thread: problems transfering databases

problems transfering databases

From
Miroslav Koncar
Date:
Hello,

I'm a newbie in the postgreSQL world, so if I'm asking what's rather
straightforward, I appologize.

This is what I'm trying to do: my collegue and I have set up three
postgreSQL databases; he has set it up on Linux RedHat 6.2, and I've
been working on RedHat 6.2 as well on Solaris 7. We are trying to
transfer all the data from his database to mine (my primary database is
on Solaris). So, we consulted the docs, and he has used the command:

pg_dumpall > backups/2001-06-04db.out

and sent me that file. But, when I try to load it to my database, I get
the error message:

obonjan[postgres-> ~/tmp]%psql -f 2001-08-21db.out
You are now connected to database template1.
SELECT
DELETE 0
DROP
'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
representation '\N
psql:2001-08-21db.out:5: PQendcopy: resetting connection

I thougth, it might be something to do with Solaris, but on Linux, like
expected, the same error.

What amd I doing wrong? Please, contact me directly, since I'm not on
the mailing list.

Thanks,
    Miroslav


Re: problems transfering databases

From
Tom Lane
Date:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connection

Given the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data.  When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.

Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?

            regards, tom lane

Re: problems transfering databases

From
Miroslav Koncar
Date:
Hello Tom,

thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:

pg_dumpall > backups/2001-06-04db.out

Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:

psql -f 2001-08-21db.out
cat 2001-06-04db.out | psql

but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.

Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).

Regards,
    Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connection

Given the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data.  When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.

Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Re: problems transfering databases

From
Tom Lane
Date:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> Since it is a simple test database, the 2001-06-04db.out file was around
> 40kB, so he has sent it to me via email.

Email huh?  I bet some part of that process was Windows?  Almost
certainly that's where the carriage return characters came from.

            regards, tom lane

Re: problems transfering databases

From
"Jeff Eckermann"
Date:
Looks like you already have the answer.  The error message is a dead giveaway.  Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'.  PostgreSQL cannot make sense out of this combination, so it returns an error.  The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql".
Edit out the CR characters, and the problem will go away.
This is what Tom already said; I am expanding on it.
----- Original Message -----
Sent: Thursday, August 23, 2001 3:18 AM
Subject: Re: [GENERAL] problems transfering databases

Hello Tom,

thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:

pg_dumpall > backups/2001-06-04db.out

Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:

psql -f 2001-08-21db.out
cat 2001-06-04db.out | psql

but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.

Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).

Regards,
    Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connection

Given the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data.  When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.

Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Problem with libpsqlodbc

From
Gilles DAROLD
Date:
Hi,

I have send some patches to the OpenLDAP team to have it work with
PostgreSQL and they should be applied asap. This works fine but with
a not usefull workaround.

I we want to have PG fully compatible with OpenLDAP we need to resolve
a problem regarding the libpsqlodbc library.

In OpenLDAP they call the function SQLBindParameter as follow:

SQLBindParameter(sth,1,SQL_PARAM_OUTPUT,SQL_C_ULONG,SQL_INTEGER,0,0,&new_keyval,0,0);

And then the statement is executed with SQLExecDirect as follow:

rc=SQLExecDirect(sth,oc->create_proc,SQL_NTS);

What they expected is that the 'new_keyval' pointer may be modified at
execution time,
with the new unique identifier returned by the query, typically a
sequence number.

It works fine with other database (mySQL, Oracle, etc.) but not with
PostgreSQL.

What's the problem ? Is the SQLBindParameter call is wrong ?
My knowledge is not enougth to find it.

If someone could explain me what's wrong !

Regards,

Gilles DAROLD



Re: problems transfering databases

From
"Rob Arnold"
Date:
I've had this problem before.  The data is stored correctly in PG, it just doesn't dump right (the CR thing I guess.  I didn't know the reason at the time).  I think this was on 7.0.x.  I worked around it by doing a pg_dump that dumped the table as "proper inserts".  That make the load MUCH slower, but it works around the issue.  Now that I know what the cause might have been, I guess I better go look at my data . . .
 
--rob
----- Original Message -----
Sent: Thursday, August 23, 2001 10:59 AM
Subject: Re: problems transfering databases

Looks like you already have the answer.  The error message is a dead giveaway.  Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'.  PostgreSQL cannot make sense out of this combination, so it returns an error.  The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql".
Edit out the CR characters, and the problem will go away.
This is what Tom already said; I am expanding on it.
----- Original Message -----
Sent: Thursday, August 23, 2001 3:18 AM
Subject: Re: [GENERAL] problems transfering databases

Hello Tom,

thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:

pg_dumpall > backups/2001-06-04db.out

Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:

psql -f 2001-08-21db.out
cat 2001-06-04db.out | psql

but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.

Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).

Regards,
    Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connection

Given the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data.  When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.

Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Re: problems transfering databases

From
Miroslav Koncar
Date:
Hello, Tom,

nope, no windows here. The file was sent direclty from the Linux to my mailbox, which I read on Solaris.

The only thing I can think of is the PostgreSQL versions. My collegue is using 7.0.3, and I use 7.1.2. Comments?

Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> Since it is a simple test database, the 2001-06-04db.out file was around
> 40kB, so he has sent it to me via email.

Email huh?  I bet some part of that process was Windows?  Almost
certainly that's where the carriage return characters came from.

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Re: problems transfering databases

From
Tom Lane
Date:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> The only thing I can think of is the PostgreSQL versions. My collegue is
> using 7.0.3, and I use 7.1.2. Comments?

That wouldn't produce a carriage-return problem.

It's possible that we're barking up the wrong tree, and that the issue
is not carriage returns but something else --- I just jumped to that
conclusion based on the way you presented the error message.  But maybe
the error message was mangled because you were sloppy about cutting-and-
pasting it into your email, or some such.  Have you looked at the data
to try to narrow down exactly which line is being rejected?

            regards, tom lane

Re: problems transfering databases

From
"Jeff Eckermann"
Date:
I very much doubt that the data wouldn't "dump right".  The dump will include any data that is in the table, including CRs if they are in there.
In the current case, the CRs were not already in there, because PostgreSQL would not have recognized that field as null ("\N") if they were.
Any transfer of a file via a Windows machine is apt to cause line endings to be silently changed, which would account for this case.  The real danger is where the rightmost field is a text type, because the CRs would be silently imported into the new installation, where they could cause real problems with any app that tries to use those fields, and a headache to debug.
----- Original Message -----
Sent: Thursday, August 23, 2001 6:32 PM
Subject: Re: problems transfering databases

I've had this problem before.  The data is stored correctly in PG, it just doesn't dump right (the CR thing I guess.  I didn't know the reason at the time).  I think this was on 7.0.x.  I worked around it by doing a pg_dump that dumped the table as "proper inserts".  That make the load MUCH slower, but it works around the issue.  Now that I know what the cause might have been, I guess I better go look at my data . . .
 
--rob
----- Original Message -----
Sent: Thursday, August 23, 2001 10:59 AM
Subject: Re: problems transfering databases

Looks like you already have the answer.  The error message is a dead giveaway.  Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'.  PostgreSQL cannot make sense out of this combination, so it returns an error.  The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql".
Edit out the CR characters, and the problem will go away.
This is what Tom already said; I am expanding on it.
----- Original Message -----
Sent: Thursday, August 23, 2001 3:18 AM
Subject: Re: [GENERAL] problems transfering databases

Hello Tom,

thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:

pg_dumpall > backups/2001-06-04db.out

Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:

psql -f 2001-08-21db.out
cat 2001-06-04db.out | psql

but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.

Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).

Regards,
    Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connection

Given the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data.  When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.

Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Re: problems transfering databases

From
"Jeff Eckermann"
Date:
I think we may be using too many words in this discussion.
Perhaps there is a message in this thread that I have not seen.
Miroslav, have you tested for carriage returns in your data, and if so, what
did you find?
If that is the problem, the carriage returns are easily edited out, and
speculation about the cause may use more time than it is worth...
If that is not the problem, we definitely have something weird.
The original error message shows the problem occurring at the first line of
your data.  Perhaps you could pipe the first few lines through some utility
that shows nonprinting characters (like "vis -lw"), and see what that shows?

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Miroslav Koncar" <miroslav.koncar@etk.ericsson.se>
Cc: <pgsql-general@postgresql.org>
Sent: Friday, August 24, 2001 8:13 AM
Subject: Re: [GENERAL] problems transfering databases


> Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> > The only thing I can think of is the PostgreSQL versions. My collegue is
> > using 7.0.3, and I use 7.1.2. Comments?
>
> That wouldn't produce a carriage-return problem.
>
> It's possible that we're barking up the wrong tree, and that the issue
> is not carriage returns but something else --- I just jumped to that
> conclusion based on the way you presented the error message.  But maybe
> the error message was mangled because you were sloppy about cutting-and-
> pasting it into your email, or some such.  Have you looked at the data
> to try to narrow down exactly which line is being rejected?
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>
>


Re: problems transfering databases

From
Rob Arnold
Date:
I never transfered the file anywhere, it was all done under linux.  The
bottom line is pg_dump follwed by a psql import failed exactly like what was
described in this thread.  When I did a dump with "proper inserts" it worked
fine.  As I said, I don't know what was causing the problem, just that the CR
thing sounded plausible.  I only do dumps with "proper inserts" now since I
know that this problem can occur and that "proper inserts" is a workaround.
I'm just confirming the behavior that someone else reported.

--rob

Jeff Eckermann <jeckermann@verio.net> said:

> I very much doubt that the data wouldn't "dump right".  The dump will
include any data that is in the table, including CRs if they are in there.
> In the current case, the CRs were not already in there, because PostgreSQL
would not have recognized that field as null ("\N") if they were.
> Any transfer of a file via a Windows machine is apt to cause line endings
to be silently changed, which would account for this case.  The real danger
is where the rightmost field is a text type, because the CRs would be
silently imported into the new installation, where they could cause real
problems with any app that tries to use those fields, and a headache to debug.
>   ----- Original Message -----
>   From: Rob Arnold
>   To: Jeff Eckermann ; Miroslav Koncar
>   Cc: pgsql-general@postgresql.org
>   Sent: Thursday, August 23, 2001 6:32 PM
>   Subject: Re: problems transfering databases
>
>
>   I've had this problem before.  The data is stored correctly in PG, it
just doesn't dump right (the CR thing I guess.  I didn't know the reason at
the time).  I think this was on 7.0.x.  I worked around it by doing a pg_dump
that dumped the table as "proper inserts".  That make the load MUCH slower,
but it works around the issue.  Now that I know what the cause might have
been, I guess I better go look at my data . . .
>
>   --rob
>     ----- Original Message -----
>     From: Jeff Eckermann
>     To: Miroslav Koncar
>     Cc: pgsql-general@postgresql.org
>     Sent: Thursday, August 23, 2001 10:59 AM
>     Subject: Re: problems transfering databases
>
>
>     Looks like you already have the answer.  The error message is a dead
giveaway.  Some field, in this case having a null value (represented as '\N')
contains a carriage return character, like: '\NCR'.  PostgreSQL cannot make
sense out of this combination, so it returns an error.  The CR contained in
the error message causes the output to return to the start of the line,
outputting the rest of the message from the beginning: see how the final " '
" overwrites the "p" in "psql".
>     Edit out the CR characters, and the problem will go away.
>     This is what Tom already said; I am expanding on it.
>       ----- Original Message -----
>       From: Miroslav Koncar
>       To: Tom Lane
>       Cc: pgsql-general@postgresql.org
>       Sent: Thursday, August 23, 2001 3:18 AM
>       Subject: Re: [GENERAL] problems transfering databases
>
>
>       Hello Tom,
>       thanks for the answer. This is the procedure, how we tried to
transfer the data from one machine to another.
>       My collegue has used the pg_dumpall command, in the following way:
>
>       pg_dumpall > backups/2001-06-04db.out
>
>       Since it is a simple test database, the 2001-06-04db.out file was
around 40kB, so he has sent it to me via email.
>       I tried to load the data from the file in two different ways:
>
>       psql -f 2001-08-21db.out
>       cat 2001-06-04db.out | psql
>
>       but got the error like stated before. I've tried to do this on
Solaris and Linux (I've set postgreSQL up on my Linux box for testing
purposes), but the same error pops up.
>
>       Could it have something to do with the versions of the databases, and
how it was installed? The version I use is
>       psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not
sure what version did my collegue use (I'll find that out, but I think it is
7.x.x).
>
>       Regards,
>           Miroslav
>
>       Tom Lane wrote:
>
>         Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
>         > 'sql:2001-08-21db.out:5: ERROR:  copy: line 1, Bad abstime
external
>         > representation '\N
>         > psql:2001-08-21db.out:5: PQendcopy: resetting connection
>         Given the weird line-wrapping of the message, I'm going to bet that
the
>         problem is that newlines in the dump file have gotten converted to
DOS
>         format (ie, \n became \r\n), and COPY is unhappy because it sees
the \r
>         as part of the data.  When it echoes the data it didn't like, the
>         presence of the \r messes up the format of the error message.
>
>         Not sure *how* that happened in a Linux-to-Solaris transfer,
though;
>         what tool did you use to transfer the dump file?
>
>                                 regards, tom lane
>
> --
> Miroslav Koncar
> Software Engineer
> Ericsson Nikola Tesla
> ETK/D/R
> Tel: +385 1 365 3479
> Fax: +385 1 365 3548
> mailto:miroslav.koncar@etk.ericsson.se
>
>




Re: problems transfering databases

From
"Rob Arnold"
Date:
Have him dump the table again (if he can) using pg_dump's "proper inserts" switch turned on.  Tom, I'm not contradicting that the problem may be directly linked to CR's in the data.  I have encountered this before and I worked around it with proper inserts.  I was working in Linux and never transferred the file anywhere.
 
--rob
 
----- Original Message -----
Sent: Friday, August 24, 2001 2:23 AM
Subject: Re: problems transfering databases

Hello, Tom,

nope, no windows here. The file was sent direclty from the Linux to my mailbox, which I read on Solaris.

The only thing I can think of is the PostgreSQL versions. My collegue is using 7.0.3, and I use 7.1.2. Comments?

Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> Since it is a simple test database, the 2001-06-04db.out file was around
> 40kB, so he has sent it to me via email.

Email huh?  I bet some part of that process was Windows?  Almost
certainly that's where the carriage return characters came from.

                        regards, tom lane

-- 
Miroslav Koncar
Software Engineer
Ericsson Nikola Tesla
ETK/D/R
Tel: +385 1 365 3479
Fax: +385 1 365 3548
mailto:miroslav.koncar@etk.ericsson.se
 

Re: problems transfering databases

From
Miroslav Koncar
Date:
Hello Tom,

problem solved! The error was caused by interpreting \N as a special character, and not what postgreSQL wants it to be. I've solved it by replaceing all \N with \\N in a texteditor, which made the shell not to interpret the character.

Thanks for your suggestions and help.

Miroslav

Tom Lane wrote:

Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> The only thing I can think of is the PostgreSQL versions. My collegue is
> using 7.0.3, and I use 7.1.2. Comments?

That wouldn't produce a carriage-return problem.

It's possible that we're barking up the wrong tree, and that the issue
is not carriage returns but something else --- I just jumped to that
conclusion based on the way you presented the error message.  But maybe
the error message was mangled because you were sloppy about cutting-and-
pasting it into your email, or some such.  Have you looked at the data
to try to narrow down exactly which line is being rejected?

                        regards, tom lane