Thread: BUG #16424: COPY Command fails for CSV formath

BUG #16424: COPY Command fails for CSV formath

From
PG Bug reporting form
Date:
The following bug has been logged on the website:

Bug reference:      16424
Logged by:          Vivek Kamath
Email address:      vivek4422@gmail.com
PostgreSQL version: 10.6
Operating system:   Linux Unix
Description:

Hi,

I am copying a csv file where the fields are quoted with " and separated  by
,.  
I am using below syntax to copy data using below syntax

psql -c "COPY table_name from filename WITH (FORMAT csv)".

On Analysis I observed that any data that even number of double quotes in it
are copied successfully but fails when it has odd number of double quotes in
it.
For ex
1234,"Test "hi"  ","Test bye"   will be loaded as 1234,Test hi, Test bye in
table successfully but my expectation is to load data as 1234,Test "hi",
Test bye
where as
1234,"Test "hi  ", "Test bye"  will fail with error ERROR:  unterminated CSV
quoted field.. In this case my expectation is to load data as 1234,Test "hi,
Test bye.

Also since all this is is happening as part of the batch. I cannot
pre-process the data as the same data set needs to be loaded in Redshift DB.
Copy command works perfectly fine with this data set in redshift.

Idea is load whatever comes in between the double quotes ("......") as is in
varchar field.
Please suggest some ideas as how this can be achieved.


Re: BUG #16424: COPY Command fails for CSV formath

From
"David G. Johnston"
Date:
On Friday, May 8, 2020, PG Bug reporting form <noreply@postgresql.org> wrote:
The following bug has been logged on the website:

Bug reference:      16424
Logged by:          Vivek Kamath
Email address:      vivek4422@gmail.com
PostgreSQL version: 10.6
Operating system:   Linux Unix
Description:       

Hi,

I am copying a csv file where the fields are quoted with " and separated  by
,. 
I am using below syntax to copy data using below syntax

psql -c "COPY table_name from filename WITH (FORMAT csv)".

Replace that command with something that will handle the poorly structured csv data you are receiving.  COPY is very linear and unforgiving when it comes to accepting input so you need something that will feed copy that well-formed data it expects).

David J.