Thread: problems transfering databases
Hello, I'm a newbie in the postgreSQL world, so if I'm asking what's rather straightforward, I appologize. This is what I'm trying to do: my collegue and I have set up three postgreSQL databases; he has set it up on Linux RedHat 6.2, and I've been working on RedHat 6.2 as well on Solaris 7. We are trying to transfer all the data from his database to mine (my primary database is on Solaris). So, we consulted the docs, and he has used the command: pg_dumpall > backups/2001-06-04db.out and sent me that file. But, when I try to load it to my database, I get the error message: obonjan[postgres-> ~/tmp]%psql -f 2001-08-21db.out You are now connected to database template1. SELECT DELETE 0 DROP 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external representation '\N psql:2001-08-21db.out:5: PQendcopy: resetting connection I thougth, it might be something to do with Solaris, but on Linux, like expected, the same error. What amd I doing wrong? Please, contact me directly, since I'm not on the mailing list. Thanks, Miroslav
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes: > 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external > representation '\N > psql:2001-08-21db.out:5: PQendcopy: resetting connection Given the weird line-wrapping of the message, I'm going to bet that the problem is that newlines in the dump file have gotten converted to DOS format (ie, \n became \r\n), and COPY is unhappy because it sees the \r as part of the data. When it echoes the data it didn't like, the presence of the \r messes up the format of the error message. Not sure *how* that happened in a Linux-to-Solaris transfer, though; what tool did you use to transfer the dump file? regards, tom lane
thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:
pg_dumpall > backups/2001-06-04db.out
Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:
psql -f 2001-08-21db.out
cat 2001-06-04db.out | psql
but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.
Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).
Regards,
Miroslav
Tom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connectionGiven the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data. When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes: > Since it is a simple test database, the 2001-06-04db.out file was around > 40kB, so he has sent it to me via email. Email huh? I bet some part of that process was Windows? Almost certainly that's where the carriage return characters came from. regards, tom lane
----- Original Message -----From: Miroslav KoncarTo: Tom LaneSent: Thursday, August 23, 2001 3:18 AMSubject: Re: [GENERAL] problems transfering databasesHello Tom,thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:pg_dumpall > backups/2001-06-04db.out
Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:psql -f 2001-08-21db.out
cat 2001-06-04db.out | psqlbut got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.
Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).Regards,
MiroslavTom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connectionGiven the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data. When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
Hi, I have send some patches to the OpenLDAP team to have it work with PostgreSQL and they should be applied asap. This works fine but with a not usefull workaround. I we want to have PG fully compatible with OpenLDAP we need to resolve a problem regarding the libpsqlodbc library. In OpenLDAP they call the function SQLBindParameter as follow: SQLBindParameter(sth,1,SQL_PARAM_OUTPUT,SQL_C_ULONG,SQL_INTEGER,0,0,&new_keyval,0,0); And then the statement is executed with SQLExecDirect as follow: rc=SQLExecDirect(sth,oc->create_proc,SQL_NTS); What they expected is that the 'new_keyval' pointer may be modified at execution time, with the new unique identifier returned by the query, typically a sequence number. It works fine with other database (mySQL, Oracle, etc.) but not with PostgreSQL. What's the problem ? Is the SQLBindParameter call is wrong ? My knowledge is not enougth to find it. If someone could explain me what's wrong ! Regards, Gilles DAROLD
----- Original Message -----From: Jeff EckermannTo: Miroslav KoncarSent: Thursday, August 23, 2001 10:59 AMSubject: Re: problems transfering databasesLooks like you already have the answer. The error message is a dead giveaway. Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'. PostgreSQL cannot make sense out of this combination, so it returns an error. The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql".Edit out the CR characters, and the problem will go away.This is what Tom already said; I am expanding on it.----- Original Message -----From: Miroslav KoncarTo: Tom LaneSent: Thursday, August 23, 2001 3:18 AMSubject: Re: [GENERAL] problems transfering databasesHello Tom,thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:pg_dumpall > backups/2001-06-04db.out
Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:psql -f 2001-08-21db.out
cat 2001-06-04db.out | psqlbut got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.
Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).Regards,
MiroslavTom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connectionGiven the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data. When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
nope, no windows here. The file was sent direclty from the Linux to my mailbox, which I read on Solaris.
The only thing I can think of is the PostgreSQL versions. My collegue is using 7.0.3, and I use 7.1.2. Comments?
Miroslav
Tom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> Since it is a simple test database, the 2001-06-04db.out file was around
> 40kB, so he has sent it to me via email.Email huh? I bet some part of that process was Windows? Almost
certainly that's where the carriage return characters came from.regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes: > The only thing I can think of is the PostgreSQL versions. My collegue is > using 7.0.3, and I use 7.1.2. Comments? That wouldn't produce a carriage-return problem. It's possible that we're barking up the wrong tree, and that the issue is not carriage returns but something else --- I just jumped to that conclusion based on the way you presented the error message. But maybe the error message was mangled because you were sloppy about cutting-and- pasting it into your email, or some such. Have you looked at the data to try to narrow down exactly which line is being rejected? regards, tom lane
----- Original Message -----From: Rob ArnoldSent: Thursday, August 23, 2001 6:32 PMSubject: Re: problems transfering databasesI've had this problem before. The data is stored correctly in PG, it just doesn't dump right (the CR thing I guess. I didn't know the reason at the time). I think this was on 7.0.x. I worked around it by doing a pg_dump that dumped the table as "proper inserts". That make the load MUCH slower, but it works around the issue. Now that I know what the cause might have been, I guess I better go look at my data . . .--rob----- Original Message -----From: Jeff EckermannTo: Miroslav KoncarSent: Thursday, August 23, 2001 10:59 AMSubject: Re: problems transfering databasesLooks like you already have the answer. The error message is a dead giveaway. Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'. PostgreSQL cannot make sense out of this combination, so it returns an error. The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql".Edit out the CR characters, and the problem will go away.This is what Tom already said; I am expanding on it.----- Original Message -----From: Miroslav KoncarTo: Tom LaneSent: Thursday, August 23, 2001 3:18 AMSubject: Re: [GENERAL] problems transfering databasesHello Tom,thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another.
My collegue has used the pg_dumpall command, in the following way:pg_dumpall > backups/2001-06-04db.out
Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email.
I tried to load the data from the file in two different ways:psql -f 2001-08-21db.out
cat 2001-06-04db.out | psqlbut got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up.
Could it have something to do with the versions of the databases, and how it was installed? The version I use is
psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x).Regards,
MiroslavTom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external
> representation '\N
> psql:2001-08-21db.out:5: PQendcopy: resetting connectionGiven the weird line-wrapping of the message, I'm going to bet that the
problem is that newlines in the dump file have gotten converted to DOS
format (ie, \n became \r\n), and COPY is unhappy because it sees the \r
as part of the data. When it echoes the data it didn't like, the
presence of the \r messes up the format of the error message.Not sure *how* that happened in a Linux-to-Solaris transfer, though;
what tool did you use to transfer the dump file?regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
I think we may be using too many words in this discussion. Perhaps there is a message in this thread that I have not seen. Miroslav, have you tested for carriage returns in your data, and if so, what did you find? If that is the problem, the carriage returns are easily edited out, and speculation about the cause may use more time than it is worth... If that is not the problem, we definitely have something weird. The original error message shows the problem occurring at the first line of your data. Perhaps you could pipe the first few lines through some utility that shows nonprinting characters (like "vis -lw"), and see what that shows? ----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Miroslav Koncar" <miroslav.koncar@etk.ericsson.se> Cc: <pgsql-general@postgresql.org> Sent: Friday, August 24, 2001 8:13 AM Subject: Re: [GENERAL] problems transfering databases > Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes: > > The only thing I can think of is the PostgreSQL versions. My collegue is > > using 7.0.3, and I use 7.1.2. Comments? > > That wouldn't produce a carriage-return problem. > > It's possible that we're barking up the wrong tree, and that the issue > is not carriage returns but something else --- I just jumped to that > conclusion based on the way you presented the error message. But maybe > the error message was mangled because you were sloppy about cutting-and- > pasting it into your email, or some such. Have you looked at the data > to try to narrow down exactly which line is being rejected? > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html > >
I never transfered the file anywhere, it was all done under linux. The bottom line is pg_dump follwed by a psql import failed exactly like what was described in this thread. When I did a dump with "proper inserts" it worked fine. As I said, I don't know what was causing the problem, just that the CR thing sounded plausible. I only do dumps with "proper inserts" now since I know that this problem can occur and that "proper inserts" is a workaround. I'm just confirming the behavior that someone else reported. --rob Jeff Eckermann <jeckermann@verio.net> said: > I very much doubt that the data wouldn't "dump right". The dump will include any data that is in the table, including CRs if they are in there. > In the current case, the CRs were not already in there, because PostgreSQL would not have recognized that field as null ("\N") if they were. > Any transfer of a file via a Windows machine is apt to cause line endings to be silently changed, which would account for this case. The real danger is where the rightmost field is a text type, because the CRs would be silently imported into the new installation, where they could cause real problems with any app that tries to use those fields, and a headache to debug. > ----- Original Message ----- > From: Rob Arnold > To: Jeff Eckermann ; Miroslav Koncar > Cc: pgsql-general@postgresql.org > Sent: Thursday, August 23, 2001 6:32 PM > Subject: Re: problems transfering databases > > > I've had this problem before. The data is stored correctly in PG, it just doesn't dump right (the CR thing I guess. I didn't know the reason at the time). I think this was on 7.0.x. I worked around it by doing a pg_dump that dumped the table as "proper inserts". That make the load MUCH slower, but it works around the issue. Now that I know what the cause might have been, I guess I better go look at my data . . . > > --rob > ----- Original Message ----- > From: Jeff Eckermann > To: Miroslav Koncar > Cc: pgsql-general@postgresql.org > Sent: Thursday, August 23, 2001 10:59 AM > Subject: Re: problems transfering databases > > > Looks like you already have the answer. The error message is a dead giveaway. Some field, in this case having a null value (represented as '\N') contains a carriage return character, like: '\NCR'. PostgreSQL cannot make sense out of this combination, so it returns an error. The CR contained in the error message causes the output to return to the start of the line, outputting the rest of the message from the beginning: see how the final " ' " overwrites the "p" in "psql". > Edit out the CR characters, and the problem will go away. > This is what Tom already said; I am expanding on it. > ----- Original Message ----- > From: Miroslav Koncar > To: Tom Lane > Cc: pgsql-general@postgresql.org > Sent: Thursday, August 23, 2001 3:18 AM > Subject: Re: [GENERAL] problems transfering databases > > > Hello Tom, > thanks for the answer. This is the procedure, how we tried to transfer the data from one machine to another. > My collegue has used the pg_dumpall command, in the following way: > > pg_dumpall > backups/2001-06-04db.out > > Since it is a simple test database, the 2001-06-04db.out file was around 40kB, so he has sent it to me via email. > I tried to load the data from the file in two different ways: > > psql -f 2001-08-21db.out > cat 2001-06-04db.out | psql > > but got the error like stated before. I've tried to do this on Solaris and Linux (I've set postgreSQL up on my Linux box for testing purposes), but the same error pops up. > > Could it have something to do with the versions of the databases, and how it was installed? The version I use is > psql (PostgreSQL) 7.1.2 (the output of psql -V command), but I'm not sure what version did my collegue use (I'll find that out, but I think it is 7.x.x). > > Regards, > Miroslav > > Tom Lane wrote: > > Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes: > > 'sql:2001-08-21db.out:5: ERROR: copy: line 1, Bad abstime external > > representation '\N > > psql:2001-08-21db.out:5: PQendcopy: resetting connection > Given the weird line-wrapping of the message, I'm going to bet that the > problem is that newlines in the dump file have gotten converted to DOS > format (ie, \n became \r\n), and COPY is unhappy because it sees the \r > as part of the data. When it echoes the data it didn't like, the > presence of the \r messes up the format of the error message. > > Not sure *how* that happened in a Linux-to-Solaris transfer, though; > what tool did you use to transfer the dump file? > > regards, tom lane > > -- > Miroslav Koncar > Software Engineer > Ericsson Nikola Tesla > ETK/D/R > Tel: +385 1 365 3479 > Fax: +385 1 365 3548 > mailto:miroslav.koncar@etk.ericsson.se > >
----- Original Message -----From: Miroslav KoncarTo: Tom LaneSent: Friday, August 24, 2001 2:23 AMSubject: Re: problems transfering databasesHello, Tom,nope, no windows here. The file was sent direclty from the Linux to my mailbox, which I read on Solaris.
The only thing I can think of is the PostgreSQL versions. My collegue is using 7.0.3, and I use 7.1.2. Comments?
Miroslav
Tom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> Since it is a simple test database, the 2001-06-04db.out file was around
> 40kB, so he has sent it to me via email.Email huh? I bet some part of that process was Windows? Almost
certainly that's where the carriage return characters came from.regards, tom lane
-- Miroslav Koncar Software Engineer Ericsson Nikola Tesla ETK/D/R Tel: +385 1 365 3479 Fax: +385 1 365 3548 mailto:miroslav.koncar@etk.ericsson.se
problem solved! The error was caused by interpreting \N as a special character, and not what postgreSQL wants it to be. I've solved it by replaceing all \N with \\N in a texteditor, which made the shell not to interpret the character.
Thanks for your suggestions and help.
Miroslav
Tom Lane wrote:
Miroslav Koncar <miroslav.koncar@etk.ericsson.se> writes:
> The only thing I can think of is the PostgreSQL versions. My collegue is
> using 7.0.3, and I use 7.1.2. Comments?That wouldn't produce a carriage-return problem.
It's possible that we're barking up the wrong tree, and that the issue
is not carriage returns but something else --- I just jumped to that
conclusion based on the way you presented the error message. But maybe
the error message was mangled because you were sloppy about cutting-and-
pasting it into your email, or some such. Have you looked at the data
to try to narrow down exactly which line is being rejected?regards, tom lane