Re: Bulk INSERT performance in 7.4.1 - Mailing list pgsql-performance

From Florian Weimer
Subject Re: Bulk INSERT performance in 7.4.1
Date
Msg-id 20040304100608.GA5521@deneb.enyo.de
Whole thread Raw
In response to Re: Bulk INSERT performance in 7.4.1  (Vivek Khera <khera@kcilink.com>)
List pgsql-performance
Vivek Khera wrote:

> If you've got the time, could you try also doing the full bulk insert
> test with the checkpoint log files on another physical disk?  See if
> that's any faster.

We have been doing that for a few weeks, but the performance
improvements are less than what we expected.  There is hardly any disk
activity on the log RAID, even during checkpointing.

After I activated the tuned configuration, we are again mostly CPU-bound
(it seems that updating all four indices is quite expensive).  The
bulk INSERT process runs single-threaded right now, and if we switched
to multiple processes for that, we could reach some 1,500 INSERTs per
second, I believe.  This is more than sufficient for us; our real-time
data collector is tuned to emit about 150 records per second, on the
average.  (There is an on-disk queue to compensate temporary problems,
such as spikes in the data rate and database updates gone awry.)

--
Current mail filters: many dial-up/DSL/cable modem hosts, and the
following domains: atlas.cz, bigpond.com, freenet.de, hotmail.com,
libero.it, netscape.net, postino.it, tiscali.co.uk, tiscali.cz,
tiscali.it, voila.fr, wanadoo.fr, yahoo.com.

pgsql-performance by date:

Previous
From: Christopher Kings-Lynne
Date:
Subject: Re: Feature request: smarter use of conditional indexes
Next
From: "Anjan Dave"
Date:
Subject: Re: Scaling further up