Re: Add new COPY option REJECT_LIMIT - Mailing list pgsql-hackers

From David G. Johnston
Subject Re: Add new COPY option REJECT_LIMIT
Date
Msg-id CAKFQuwYP91_G6tktYFTZq_CmkZ_=zuWjkz1+25Nd8bpsrDkx5Q@mail.gmail.com
Whole thread Raw
In response to Add new COPY option REJECT_LIMIT  (torikoshia <torikoshia@oss.nttdata.com>)
Responses Re: Add new COPY option REJECT_LIMIT
List pgsql-hackers
On Fri, Jan 26, 2024 at 2:49 AM torikoshia <torikoshia@oss.nttdata.com> wrote:
Hi,

9e2d870 enabled the COPY command to skip soft error, and I think we can
add another option which specifies the maximum tolerable number of soft
errors.

I remember this was discussed in [1], and feel it would be useful when
loading 'dirty' data but there is a limit to how dirty it can be.

Attached a patch for this.

What do you think?


I'm opposed to adding this particular feature.

When implementing this kind of business rule I'd need the option to specify a percentage, not just an absolute value.

I would focus on trying to put the data required to make this kind of determination into a place where applications implementing such business rules and monitoring can readily get at it.  The "ERRORS TO" and maybe a corresponding "STATS TO" option where a table can be specified for the system to place the problematic data and stats about the copy itself.

David J.

pgsql-hackers by date:

Previous
From: vignesh C
Date:
Subject: Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning
Next
From: Tom Lane
Date:
Subject: Re: POC: GROUP BY optimization