Re: [psycopg] speed concerns with executemany() - Mailing list psycopg

From Daniele Varrazzo
Subject Re: [psycopg] speed concerns with executemany()
Date
Msg-id CA+mi_8aFtNS3jN6OTn00kcLD8tGBcN61dMDxj3pAaP5xvG1Svg@mail.gmail.com
Whole thread Raw
In response to Re: [psycopg] speed concerns with executemany()  (Christophe Pettus <xof@thebuild.com>)
Responses Re: [psycopg] speed concerns with executemany()  (Karsten Hilbert <Karsten.Hilbert@gmx.net>)
List psycopg
On Mon, Jan 2, 2017 at 5:16 PM, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
> On Mon, Jan 02, 2017 at 05:10:10PM +0100, Daniele Varrazzo wrote:
>
>> > Any chance 'l' can be supported to be a generator expression ?
>>
>> The function would consume only page_size records at time. This is to
>> avoid creating a humongous query in memory and then on the server, but
>> has the nice effect of requesting only as many records from the
>> sequence.
>
> Nice.
>
> Can said sequence be a generator (IOW not needing to support
> len() upfront) ?

Correct: see the implementation of paginate() in the gist at
<https://gist.github.com/dvarrazzo/4204cca5d1cd7c9c95bc814d81e6b83e>:
the sequence is only consumed via next(). Final implementation may
change but accessing the sequence only as a generator is a desired
feature for me too.

-- Daniele


psycopg by date:

Previous
From: Daniele Varrazzo
Date:
Subject: Re: [psycopg] Solving the SQL composition problem
Next
From: Karsten Hilbert
Date:
Subject: Re: [psycopg] speed concerns with executemany()