Re: Better way to bulk-load millions of CSV records into postgres? - Mailing list pgsql-novice

From Josh Berkus
Subject Re: Better way to bulk-load millions of CSV records into postgres?
Date
Msg-id 200205211639.25237.josh@agliodbs.com
Whole thread Raw
In response to Better way to bulk-load millions of CSV records into postgres?  (Ron Johnson <ron.l.johnson@cox.net>)
List pgsql-novice
Ron,

> Currently, I've got a python script using pyPgSQL that
> parses the CSV record, creates a string that is a big
> "INSERT INTO VALUES (...)" command, then, execute() it.

What's wrong with the COPY command?

> top shows that this method uses postmaster with ~70% CPU
> utilization, and python with ~15% utilization.
>
> Still, it's only inserting ~190 recs/second.  Is there a
> better way to do this, or am I constrained by the hardware?

This sounds pretty good for an ATA system.   Upgrading to SCSI-RAID will also
improve your performance.

-Josh Berkus

pgsql-novice by date:

Previous
From: Tom Lane
Date:
Subject: Re: Large tables being split at 1GB boundary
Next
From: Victor Manuel Torres Aguirre
Date:
Subject: PostgreSQL+Access97+Linux: How to..