Re: libpq - lack of support to set the fetch size - Mailing list pgsql-general

From Jeff Janes
Subject Re: libpq - lack of support to set the fetch size
Date
Msg-id CAMkU=1zhd9CDKNVm3yQmwTJxRfefZ8-DB-jjWAaFQj3GOTs+rg@mail.gmail.com
Whole thread Raw
In response to libpq - lack of support to set the fetch size  (matshyeq <matshyeq@gmail.com>)
Responses Re: libpq - lack of support to set the fetch size  ("Greg Sabino Mullane" <greg@turnstep.com>)
List pgsql-general
On Sun, Mar 9, 2014 at 6:43 AM, matshyeq <matshyeq@gmail.com> wrote:
Hello,

I've found an issue when tried to implement fetching rows from big table (2mln rows) in my app.
Basically I don't find an elegant and easy way (other than always use cursors) to limit the number of rows returned.
This causes my application to break due to the excessive memory consumption.

I'm using Perl and DBD::Pg library but contacted maintainer who actually pointed out this is an issue that goes much deeper (libpq):

"Unfortunately, this is a limitation in the underlying driver (libpq) rather than DBD::Pg itself. There have been talks over the years of supporting this, but nothing concrete yet. Your best bet would be to ask about this on the Postgres lists"

I don't think this is correct.  First, DBD::Pg could get tricky and automatically wrap your query in a cursor and then fetch from the cursor behind the scenes.  I believe that this is what Python's module does for you in some modes.  Second, the feature needed to do this without even using a cursor was added 1.5 years ago (PQsetSingleRowMode).  The DBD::Pg was just not taught how to use it yet.

The first strategy could probably be done purely in Perl, the second would require changes to the C parts of DBD::Pg.

Of course just because it can be implemented in DBD::Pg doesn't mean anyone has an obligation to do it.  You could speed that along by contributing the code yourself.  But I would say the ball is firmly in DBD::Pg's court.


Cheers,

Jeff

pgsql-general by date:

Previous
From: Thom Brown
Date:
Subject: Re: 9.1.11 - many backends in "semtimedop" syscall
Next
From: Tim Kane
Date:
Subject: Re: Playing with 9.4devel - unnest