COPY problems with psql / libpq - Mailing list pgsql-hackers

From Oliver Elphick
Subject COPY problems with psql / libpq
Date
Msg-id 200001201521.PAA04675@linda.lfix.co.uk
Whole thread Raw
Responses Re: [HACKERS] COPY problems with psql / libpq
List pgsql-hackers
[Using current cvs] I have a problem with COPY when called like this:
 psql -e bray </tmp/ol

with one particular file, containing commands and data like this:

========================================
copy address from stdin;
1  Some place, Regal Way     Somewhere Oxon AB1 3CF  [ Tel: 01367 888888 ]   NoDeliveries Pm Fridays        GB \N \N \N
\N\N \N \N \N \N \N \N \N
 
...
1000     73 Some Road      London      SW1 1ZZ  GB 44 81 999   9999     \N  \N \N \N \N \N \N \N
\.
-- 1000 records written

select count(*) from address;

copy address from stdin;
1001...
... and so on up to 3916 records in total, divided into 1000 record chunks
========================================

psql or libpq seems to choke on the data, so that some spurious error
arises, such as null input into a non-null field.  Thereafter, libpq
seems to get stuck in a COPY state:

copy address from stdin;
-- 1000 records written
select count(*) from address;
PQexec: you gotta get out of a COPY state yourself.

(The comment in PQexec says it does this to preserve backwards
compatibility, but getting stuck in COPY state is not backwards
compatible!)

If I remove the SQL commands from the input file, go into psql and do:
 copy address from '/tmp/ol';

all 3916 records are added correctly.  This seems to indicate that
the problem is not in the backend.

I found that if I broke the first 1000 records into 2 equal parts, all
of them were added correctly without error; so I conclude that data
is being buffered and lost somewhere in psql or libpq, and the problem is
dependent on the amount of data being copied.

This began to happen within the last week, but I don't know which
recent change is responsible.

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver              PGP key from public servers; key
ID32B8FAA1                ========================================    "Neither is there salvation in any other; for
thereis      none other name under heaven given among men, whereby      we must be saved."               Acts 4:12 
 




pgsql-hackers by date:

Previous
From: Thomas Lockhart
Date:
Subject: Re: [HACKERS] A notice for too long names
Next
From: Bruce Momjian
Date:
Subject: Re: [HACKERS] Re: Postgres improvement