Re: [psycopg] speed concerns with executemany() - Mailing list psycopg

From Oswaldo
Subject Re: [psycopg] speed concerns with executemany()
Date
Msg-id cdbf56fa-1f82-c60f-f3f3-7f6c636bddf5@soft-com.es
Whole thread Raw
In response to Re: [psycopg] speed concerns with executemany()  (Daniele Varrazzo <daniele.varrazzo@gmail.com>)
Responses Re: [psycopg] speed concerns with executemany()  (Oswaldo <listas@soft-com.es>)
List psycopg
El 02/01/17 a las 17:07, Daniele Varrazzo escribió:
> On Mon, Jan 2, 2017 at 4:35 PM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>>
>> With NRECS=10000 and page size=100:
>>
>> aklaver@tito:~> python psycopg_executemany.py -p 100
>> classic: 427.618795156 sec
>> joined: 7.55754685402 sec
>

Hello,

There is a third option that provides a small improvement: generate a
single sql with multple values.

- Test with local database:

classic: 1.53970813751 sec
joined: 0.564052820206 sec
joined values: 0.175103187561 sec

- Test with db on an internet server

classic: 236.342775822 sec
joined: 6.08789801598 sec
joined values: 4.49090409279 sec

I often need to move data between different internet servers (sql server
<-> Postgresql). In my experience this is the fastest way to move
hundreds of thousands of data records.

I attach the sample modified with it executemany3 function.

(Sorry for my bad english)
Regards.




psycopg by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: [psycopg] Solving the SQL composition problem
Next
From: Oswaldo
Date:
Subject: Re: [psycopg] speed concerns with executemany()