Re: COPY from STDIN vs file with large CSVs - Mailing list pgsql-admin

From bricklen
Subject Re: COPY from STDIN vs file with large CSVs
Date
Msg-id CAGrpgQ-zFD2aBz_+w4V5UWMH1wGOhjefy7bPz4S-VTcr52dusg@mail.gmail.com
Whole thread Raw
In response to COPY from STDIN vs file with large CSVs  (Wells Oliver <wells.oliver@gmail.com>)
List pgsql-admin


On Wed, Jan 8, 2020 at 8:55 AM Wells Oliver <wells.oliver@gmail.com> wrote:
I have a CSV that's ~30GB. Some 400m rows. Would there be a meaningful performance difference to run COPY from STDIN using: cat f.csv | psql "COPY .. FROM STDIN WITH CSV" versus just doing "COPY ... FROM 'f.csv' WITH CSV"?

If you're looking to speed up the loading - and your disk subsystem is decent - consider running your csv through the "split" command to decompose it into smaller CSV files. You can then load them in parallel using multiple psql sessions.

pgsql-admin by date:

Previous
From: "David G. Johnston"
Date:
Subject: Re: COPY from STDIN vs file with large CSVs
Next
From: bvo
Date:
Subject: misbehavior slave