Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) - Mailing list pgsql-hackers

From Nikita Malakhov
Subject Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
Date
Msg-id CAN-LCVPyWKAgPMPkfkoe6_H3uG1ue1a6YbK0Zsn_UEL_CM-3QA@mail.gmail.com
Whole thread Raw
In response to Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)  (Damir Belyalov <dam.bel07@gmail.com>)
List pgsql-hackers
Hi Damir!

Your work looks like a very promising feature for production systems,
where data often needs to be loaded from external sources.

I've looked over the discussion and want to make a proposal -
when we load a bunch of records in database it does not make sense
to output errors to command output, and does not make sense to limit
error output to any number at all, because if we decided to load data
anyway - we would want to have a list (a file) with all records that were
discarded because of errors, with related error information, to, say,
deal with errors and process these records later. It looks like a reasonable
addition to your patch.

As a command output some limited number of error messages has much
less meaning than overall stats - records processed, records loaded,
records discarded, total number of errors.

For example you can look the Oracle SQL Loader feature, I hope this could
give some ideas for further improvements.

On Wed, Nov 2, 2022 at 11:46 AM Damir Belyalov <dam.bel07@gmail.com> wrote:
Updated the patch:
- Optimized and simplified logic of IGNORE_ERRORS
- Changed variable names to more understandable ones
- Added an analogue of MAX_BUFFERED_BYTES for safe buffer


Regards,
Damir Belyalov
Postgres Professional


--
Regards,
Nikita Malakhov
Postgres Professional 

pgsql-hackers by date:

Previous
From: Melih Mutlu
Date:
Subject: Re: wake up logical workers after ALTER SUBSCRIPTION
Next
From: vignesh C
Date:
Subject: Re: Support logical replication of DDLs