Thread: START/END line number for COPY FROM

START/END line number for COPY FROM

From
Surafel Temesgen
Date:

Hi,

Currently we can skip header line on COPY FROM but having the ability to skip and stop copying at any line can use to divide long copy operation and enable to copy a subset of the file and skipping footer. Attach is a patch for it


Regards

Surafel

Attachment

Re: START/END line number for COPY FROM

From
Tom Lane
Date:
Surafel Temesgen <surafel3000@gmail.com> writes:
> Currently we can skip header line on COPY FROM but having the ability to
> skip and stop copying at any line can use to divide long copy operation and
> enable to copy a subset of the file and skipping footer. Attach is a patch
> for it

I do not think this is a good idea.  We have resisted attempts to add
ETL-like features to COPY on the grounds that it would add complexity
and cost performance, and that that's not what COPY is for.   This
seems to fall squarely in the domain of something you should be doing
with another tool.

            regards, tom lane


Re: START/END line number for COPY FROM

From
Peter Eisentraut
Date:
On 20/12/2018 14:02, Surafel Temesgen wrote:
> Currently we can skip header line on COPY FROM but having the ability to
> skip and stop copying at any line can use to divide long copy operation
> and enable to copy a subset of the file and skipping footer.

It seems a bit fragile to me if I want to skip a footer and need to
figure out the total line count, subtract one, and then oh, was it zero-
or one-based.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Re: START/END line number for COPY FROM

From
Surafel Temesgen
Date:

Hi,
On Fri, Jan 4, 2019 at 5:37 PM Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
It seems a bit fragile to me if I want to skip a footer and need to
figure out the total line count, subtract one, and then oh, was it zero-
or one-based.


But normally we don't say start copying from line number 0
regards
Surafel

Re: START/END line number for COPY FROM

From
David Rowley
Date:
On Fri, 21 Dec 2018 at 02:02, Surafel Temesgen <surafel3000@gmail.com> wrote:
> Currently we can skip header line on COPY FROM but having the ability to skip and stop copying at any line can use to
dividelong copy operation and enable to copy a subset of the file and skipping footer. Attach is a patch for it
 

I'm struggling a bit to see the sense in this. If you really want to
improve the performance of a long copy, then I think it makes more
sense to have performed the backup in multiple pieces in the first
place.  Having the database read the input stream and ignore the first
N lines sounds like a bit of a waste of effort, and effort that
wouldn't be required if the COPY TO had been done in multiple pieces.

-- 
 David Rowley                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: START/END line number for COPY FROM

From
Surafel Temesgen
Date:

Hi,
On Sat, Jan 5, 2019 at 1:10 PM David Rowley <david.rowley@2ndquadrant.com> wrote:
On Fri, 21 Dec 2018 at 02:02, Surafel Temesgen <surafel3000@gmail.com> wrote:
> Currently we can skip header line on COPY FROM but having the ability to skip and stop copying at any line can use to divide long copy operation and enable to copy a subset of the file and skipping footer. Attach is a patch for it

I'm struggling a bit to see the sense in this. If you really want to
improve the performance of a long copy, then I think it makes more
sense to have performed the backup in multiple pieces in the first
place. 


it is not always the case to have in control of the data importing it may came from
external system

regards
Surafel

Re: START/END line number for COPY FROM

From
Peter Eisentraut
Date:
On 06/01/2019 12:59, Surafel Temesgen wrote:
> it is not always the case to have in control of the data importing it
> may came from
> external system

But the problem that David described remains:  If your data loading
requirement is so complicated that you need to load the file in chunks,
then doing it by line numbers will require you to skip over the leading
lines at every subsequent chunk.  That's not going to be good for larger
files.

I think your problem needs a different solution.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services