Re: large numbers of inserts out of memory strategy - Mailing list pgsql-general

From Steven Lembark
Subject Re: large numbers of inserts out of memory strategy
Date
Msg-id 20171129110210.59a57d4d@wrkhors.com
Whole thread Raw
In response to Re: large numbers of inserts out of memory strategy  (Ted Toth <txtoth@gmail.com>)
List pgsql-general
> > what tools / languages ate you using?  
> 
> I'm using python to read binary source files and create the text files
> contains the SQL. Them I'm running psql -f <file containing SQL>.

Then chunking the input should be trivial.
There are a variety of techniques you can use to things like disable
indexes during loading, etc. Maybe load them into temp tables and 
then insert the temp's into the destination tables. The point is to 
amortize the memory load over the entire load period.

-- 
Steven Lembark                                       1505 National Ave
Workhorse Computing                                 Rockford, IL 61103
lembark@wrkhors.com                                    +1 888 359 3508


pgsql-general by date:

Previous
From: Nicola Contu
Date:
Subject: pg_replication_slots
Next
From: Steven Lembark
Date:
Subject: Re: large numbers of inserts out of memory strategy