Re: performance for high-volume log insertion - Mailing list pgsql-performance

From Glenn Maynard
Subject Re: performance for high-volume log insertion
Date
Msg-id bd36f99e0904221420v3f8680c6m65033c4738efa179@mail.gmail.com
Whole thread Raw
In response to Re: performance for high-volume log insertion  (Stephen Frost <sfrost@snowman.net>)
Responses Re: performance for high-volume log insertion
List pgsql-performance
On Wed, Apr 22, 2009 at 4:37 PM, Stephen Frost <sfrost@snowman.net> wrote:
> Thanks for doing the work.  I had been intending to but hadn't gotten to
> it yet.

I'd done similar tests recently, for some batch import code, so it was
just a matter of recreating it.

>> separate inserts, no transaction: 21.21s
>> separate inserts, same transaction: 1.89s
>> 40 inserts, 100 rows/insert: 0.18s
>> one 40000-value insert: 0.16s
>> 40 prepared inserts, 100 rows/insert: 0.15s
>> COPY (text): 0.10s
>> COPY (binary): 0.10s
>
> What about 40000 individual prepared inserts?  Just curious about it.

40000 inserts, one prepared statement each (constructing the prepared
statement only once), in a single transaction: 1.68s

I'm surprised that there's any win here at all.

> Also, kind of suprised about COPY text vs. binary.  What was the data
> type in the table..?  If text, that makes sense, if it was an integer or
> something else, I'm kind of suprised.

Each row had one integer column.  I expect strings to be harder to
parse, since it's allocating buffers and parsing escapes, which is
usually more expensive than parsing an integer out of a string.  I'd
expect the difference to be negligible either way, though, and I'd be
interested in hearing about use cases where binary copying is enough
of a win to be worth the development cost of maintaining the feature.

On Wed, Apr 22, 2009 at 4:49 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Also, just to be clear: were the "40 insert" cases 40 separate
> transactions or one transaction?  I think the latter was meant but
> it's not 100% clear.

One transaction--the only case where I ran more than one transaction
was the first.

--
Glenn Maynard

pgsql-performance by date:

Previous
From: "Joshua D. Drake"
Date:
Subject: Re: performance for high-volume log insertion
Next
From: Glenn Maynard
Date:
Subject: Re: performance for high-volume log insertion