Postgres Pro
Downloads
Home   >   mailing lists

Why "copy ... from stdio" does not return immediately when reading invalid data? - Mailing list pgsql-general

From Nicolas Grilly
Subject Why "copy ... from stdio" does not return immediately when reading invalid data?
Date
Msg-id AANLkTik9uhtpp2bthp=aXt70VHPFCSCR8ma3NDnkkwKc@mail.gmail.com
Whole thread Raw
Responses Re: Why "copy ... from stdio" does not return immediately when reading invalid data?
List pgsql-general
Tree view
Hello,

I am importing gigabytes of data into PostgreSQL, and I don't want to wait 10 minutes just to discover an error in the 10th line of my input file.

I tried the command "\copy ... from stdio" in psql and it looks like psql has to read the entire input before returning a potential error, even if the invalid value is in one of the first rows.

Is it a limitation of PostgreSQL protocol, of the library lipq, or of the tool psql?

Is the copy protocol (aka PQputCopyData and PQputCopyEnd) designed to send gigabytes of data with just one "copy ... from stdio" query, and is there a way to be notified of a potential error before calling PQputCopyEnd? Or do I have to send my data in small chunks (for example batch of 10000 rows), issue a PQputCopyEnd, check for errors, and continue with the next chunk?

Thanks for your help and advice.

Regards,

Nicolas Grilly

pgsql-general by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: Streaming Rep 101 questions
Next
From: Chris Browne
Date:
Subject: Re: Why does my DB size differ between Production and DR? (Postgres 8.4)