Re: Why "copy ... from stdio" does not return immediately when reading invalid data? - Mailing list pgsql-general

From John R Pierce
Subject Re: Why "copy ... from stdio" does not return immediately when reading invalid data?
Date
Msg-id 4D49AB19.4020502@hogranch.com
Whole thread Raw
In response to Why "copy ... from stdio" does not return immediately when reading invalid data?  (Nicolas Grilly <nicolas@gardentechno.com>)
Responses Re: Why "copy ... from stdio" does not return immediately when reading invalid data?  (Nicolas Grilly <nicolas@gardentechno.com>)
List pgsql-general
On 02/02/11 10:20 AM, Nicolas Grilly wrote:
> Is the copy protocol (aka PQputCopyData and PQputCopyEnd) designed to
> send gigabytes of data with just one "copy ... from stdio" query, and
> is there a way to be notified of a potential error before calling
> PQputCopyEnd? Or do I have to send my data in small chunks (for
> example batch of 10000 rows), issue a PQputCopyEnd, check for errors,
> and continue with the next chunk?

I would batch the data, maybe 1000 lines or even 100 lines at a time if
these errors are at all frequent.  put the errored batches in an
exception list or something so you can sort them out later.



pgsql-general by date:

Previous
From: Carlos Mennens
Date:
Subject: Database Design Question
Next
From: John R Pierce
Date:
Subject: Re: Database Design Question