Re: [HACKERS] GSOC'17 project introduction: Parallel COPY executionwith errors handling - Mailing list pgsql-hackers

From Alex K
Subject Re: [HACKERS] GSOC'17 project introduction: Parallel COPY executionwith errors handling
Date
Msg-id CADfU8Wy3ZSSdkWpBsW-rkXR3OSg+LsWiGP3PEEN8dapiA7-x-w@mail.gmail.com
Whole thread Raw
In response to Re: [HACKERS] GSOC'17 project introduction: Parallel COPY executionwith errors handling  (Craig Ringer <craig@2ndquadrant.com>)
Responses Re: [HACKERS] GSOC'17 project introduction: Parallel COPY executionwith errors handling  (Peter Geoghegan <pg@bowt.ie>)
List pgsql-hackers
Hi pgsql-hackers,

Thank you again for all these replies. I have started working under this project 
and learnt a lot of new stuff last month, so here are some new thoughts about 
ERRORS handling in COPY. I decided to stick to the same thread, since it 
has a neutral subject.

(1) One of my mentors--Alvaro Herrera--suggested me to have a look on the 
UPSERT. It may be a good point to be able to achieve the same functionality 
as during the ON CONFLICT DO NOTHING, when COPY actually inserts tuples 
and errors handling is turned on. It could additionally reduce number of failed 
subtransactions and reduce XIDs consumption, while still ignoring some common 
errors like unique index violation.

Adding a full support of ON CONFLICT DO NOTHING/UPDATE to COPY seems 
to be a large separated task and is out of the current project scope, but maybe there is 
a relatively simple way to somehow perform internally tuples insert with 
ON CONFLICT DO NOTHING? I have added Peter Geoghegan to cc, as 
I understand he is the major contributor of UPSERT in PostgreSQL. It would be great
if he will answer this question.

(2) Otherwise, I am still going to use subtransactions via BeginInternalSubTransactio
and PG_TRY / PG_CATCH with 
ReleaseCurrentSubTransaction / RollbackAndReleaseCurrentSubTransaction. 
To minimize XIDs consumption I will try to insert tuples in batches and pre-validate 
them as much as possible (as was suggested in the thread before).



Alexey



pgsql-hackers by date:

Previous
From: "David G. Johnston"
Date:
Subject: Re: [HACKERS] PostgreSQL 10 changes in exclusion constraints - didsomething change? CASE WHEN behavior oddity
Next
From: Robert Haas
Date:
Subject: Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw