Postgres Pro
Downloads
Home   >   mailing lists

Re: [psycopg] speed concerns with executemany() - Mailing list psycopg

From Aryeh Leib Taurog
Subject Re: [psycopg] speed concerns with executemany()
Date
Msg-id 20170129174410.GC2716@deb76.aryehleib.com
Whole thread Raw
In response to Re: [psycopg] speed concerns with executemany()  (Aryeh Leib Taurog <python@aryehleib.com>)
List psycopg
Tree view 
On Thu, Jan 19, 2017 at 02:23:15PM +0200, Aryeh Leib Taurog wrote:
> >>
> >> Adrian, have you got a benchmark "classic vs. joined" on remote
> >> network? Thank you.
> >
> > With NRECS=10000 and page size=100:
> >
> > aklaver(at)tito:~> python psycopg_executemany.py -p 100
> > classic: 427.618795156 sec
> > joined: 7.55754685402 sec
>
> This is really interesting.  I have long been using a utility I put
> together to insert using BINARY COPY.  In fact I just brushed it up a
> bit and put it on PyPi: <https://pypi.python.org/pypi/pgcopy>

I added binary copy to the benchmark:
<https://gist.github.com/altaurog/84668e034646fb354b5de81bb86a580d>

On local machine, with NRECS=100000 and page size=100:
classic: 4.26264309883 sec
joined: 2.34096288681 sec
pgcopy: 0.412513971329 sec


Over network, with NRECS=10000 and page size=100:
classic: 716.759769917 sec
joined: 15.141461134 sec
pgcopy: 3.70594096184 sec

psycopg by date:

Previous
From: Jonathan Rogers
Date:
Subject: Re: [psycopg] Nested transactions support for code composability
Next
From: Daniele Varrazzo
Date:
Subject: Re: [psycopg] speed concerns with executemany()