Thread: About large objects asynchronous and non-blocking support

About large objects asynchronous and non-blocking support

From
Giovanni Mascellani
Date:
Hi.

At the moment libpq doesn't seem to support asynchronous and
non-blocking support for large objects, in the style of
PQsendQuery/PQgetResult. This makes large objects hardly suited for
single-threaded programs based on some variant of select().

I would like to know whether this is a deliberate decision or it is
considered a bug, and, in case, whether it is scheduled to be fixed.

Though I cannot guarantee anything, I may be interested into working out
a patch, if no one is already doing the same (of course I understand
that this patch wouldn't be for 9.3, which is already in its late
release cycle).

Do you think this may be of interest?

Thanks, Giovanni.
--
Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
Pisa, Italy

Web: http://poisson.phc.unipi.it/~mascellani
Jabber: g.mascellani@jabber.org / giovanni@elabor.homelinux.org


Re: About large objects asynchronous and non-blocking support

From
Dmitriy Igrishin
Date:



2013/6/5 Giovanni Mascellani <g.mascellani@gmail.com>
Hi.

At the moment libpq doesn't seem to support asynchronous and
non-blocking support for large objects, in the style of
PQsendQuery/PQgetResult. This makes large objects hardly suited for
single-threaded programs based on some variant of select().
"There are server-side functions callable from SQL that correspond to each of
the client-side functions". Hence, you can call these functions by using
asynchronous API.

--
// Dmitriy.

Re: About large objects asynchronous and non-blocking support

From
Tatsuo Ishii
Date:
> Hi.
> 
> At the moment libpq doesn't seem to support asynchronous and
> non-blocking support for large objects, in the style of
> PQsendQuery/PQgetResult. This makes large objects hardly suited for
> single-threaded programs based on some variant of select().
> 
> I would like to know whether this is a deliberate decision or it is
> considered a bug, and, in case, whether it is scheduled to be fixed.

Certainly not bug, since the doc clearly stats that PQsendQuery can
only be used as a substituation of PQexec.  (see "Asynchronous Command
Processing" section" for more details). The large object API is
completely different from PQexec and its friends, so it cannot be used
with PQsendQuery.

Talking about more details, PQexec and PQsendQuery is designed to
handle only "Q" messsage out of PostgreSQL frontend/backend protocol,
while to access large objects, you need to handle "V" message.

> Though I cannot guarantee anything, I may be interested into working out
> a patch, if no one is already doing the same (of course I understand
> that this patch wouldn't be for 9.3, which is already in its late
> release cycle).
> 
> Do you think this may be of interest?

Yes, I understand your pain, and I myself think we need new APIs for
large objects. Probably that would be not terribly hard. One idea
would be inventing an asynchronous version of PQfn and let
lo_read/lo_write allow to use the new API.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp



Re: About large objects asynchronous and non-blocking support

From
Dmitriy Igrishin
Date:
<div dir="ltr"><br /><div class="gmail_extra"><br /><br /><div class="gmail_quote">2013/6/6 Tatsuo Ishii <span
dir="ltr"><<ahref="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span><br /><blockquote
class="gmail_quote"style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div
class="im">>Hi.<br /> ><br /> > At the moment libpq doesn't seem to support asynchronous and<br /> >
non-blockingsupport for large objects, in the style of<br /> > PQsendQuery/PQgetResult. This makes large objects
hardlysuited for<br /> > single-threaded programs based on some variant of select().<br /> ><br /> > I would
liketo know whether this is a deliberate decision or it is<br /> > considered a bug, and, in case, whether it is
scheduledto be fixed.<br /><br /></div>Certainly not bug, since the doc clearly stats that PQsendQuery can<br /> only
beused as a substituation of PQexec.  (see "Asynchronous Command<br /> Processing" section" for more details). The
largeobject API is<br /> completely different from PQexec and its friends, so it cannot be used<br /> with
PQsendQuery.<br/><br /> Talking about more details, PQexec and PQsendQuery is designed to<br /> handle only "Q"
messsageout of PostgreSQL frontend/backend protocol,<br /> while to access large objects, you need to handle "V"
message.<br/></blockquote><div style="style">Really? I've specialized a C++ standard std::streambuf class by
using</div><divstyle="style">only extended query protocol (by using prepared statements via PQsendPrepare,</div><div
style="style">PQsendQueryPrepared)to call SQL functions like loread(), lowrite(), lo_tell(), etc.</div><div
style="style">Allthese functions just needs to be called inside BEGIN block. And yes,</div><div style="style">it can be
doneasynchronously.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div
class="im"><br/> > Though I cannot guarantee anything, I may be interested into working out<br /> > a patch, if
noone is already doing the same (of course I understand<br /> > that this patch wouldn't be for 9.3, which is
alreadyin its late<br /> > release cycle).<br /> ><br /> > Do you think this may be of interest?<br /><br
/></div>Yes,I understand your pain, and I myself think we need new APIs for<br /> large objects. Probably that would be
notterribly hard. One idea<br /> would be inventing an asynchronous version of PQfn and let<br /> lo_read/lo_write
allowto use the new API.<br /></blockquote><div style="style">Yes, but according to <a
href="http://www.postgresql.org/docs/9.2/static/protocol-flow.html#AEN95330">http://www.postgresql.org/docs/9.2/static/protocol-flow.html#AEN95330</a></div><div
style="style">and/or <a
href="http://www.postgresql.org/docs/9.2/static/libpq-fastpath.html">http://www.postgresql.org/docs/9.2/static/libpq-fastpath.html</a></div><div
style="style">functioncall sub-protocol is obsolete. Thats why personally I decided to</div><div style="style">use
preparedstatements.</div></div><br />-- <br />// Dmitriy.<br /><br /></div></div> 

Re: About large objects asynchronous and non-blocking support

From
Giovanni Mascellani
Date:
Hi.

Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto:
>> At the moment libpq doesn't seem to support asynchronous and
>> non-blocking support for large objects, in the style of
>> PQsendQuery/PQgetResult. This makes large objects hardly suited for
>> single-threaded programs based on some variant of select().
>>
> According to http://www.postgresql.org/docs/9.2/static/lo-funcs.html
> "There are server-side functions callable from SQL that correspond to each
> of
> the client-side functions". Hence, you can call these functions by using
> asynchronous API.

Thanks, I'll try this way (BTW, it may help to specify on the
documentation that lo_read and lo_write lose the "_"). I wonder whether
having to escape all the content for lowrite  can't have a negative
impact on performances. It shouldn't be too bad for my case, though.

Giovanni.
--
Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
Pisa, Italy

Web: http://poisson.phc.unipi.it/~mascellani
Jabber: g.mascellani@jabber.org / giovanni@elabor.homelinux.org


Re: About large objects asynchronous and non-blocking support

From
Dmitriy Igrishin
Date:



2013/6/8 Giovanni Mascellani <g.mascellani@gmail.com>
Hi.

Il 05/06/2013 22:52, Dmitriy Igrishin ha scritto:
>> At the moment libpq doesn't seem to support asynchronous and
>> non-blocking support for large objects, in the style of
>> PQsendQuery/PQgetResult. This makes large objects hardly suited for
>> single-threaded programs based on some variant of select().
>>
> According to http://www.postgresql.org/docs/9.2/static/lo-funcs.html
> "There are server-side functions callable from SQL that correspond to each
> of
> the client-side functions". Hence, you can call these functions by using
> asynchronous API.

Thanks, I'll try this way (BTW, it may help to specify on the
documentation that lo_read and lo_write lose the "_"). I wonder whether
having to escape all the content for lowrite  can't have a negative
impact on performances. It shouldn't be too bad for my case, though.
You may avoid escaping bytea data by using PQsendPrepare,
PQsendQueryPrepared specifying binary data format.

--
// Dmitriy.

Re: About large objects asynchronous and non-blocking support

From
Tatsuo Ishii
Date:
> 2013/6/6 Tatsuo Ishii <ishii@postgresql.org>
> 
>> > Hi.
>> >
>> > At the moment libpq doesn't seem to support asynchronous and
>> > non-blocking support for large objects, in the style of
>> > PQsendQuery/PQgetResult. This makes large objects hardly suited for
>> > single-threaded programs based on some variant of select().
>> >
>> > I would like to know whether this is a deliberate decision or it is
>> > considered a bug, and, in case, whether it is scheduled to be fixed.
>>
>> Certainly not bug, since the doc clearly stats that PQsendQuery can
>> only be used as a substituation of PQexec.  (see "Asynchronous Command
>> Processing" section" for more details). The large object API is
>> completely different from PQexec and its friends, so it cannot be used
>> with PQsendQuery.
>>
>> Talking about more details, PQexec and PQsendQuery is designed to
>> handle only "Q" messsage out of PostgreSQL frontend/backend protocol,
>> while to access large objects, you need to handle "V" message.
>>
> Really? I've specialized a C++ standard std::streambuf class by using
> only extended query protocol (by using prepared statements via
> PQsendPrepare,
> PQsendQueryPrepared) to call SQL functions like loread(), lowrite(),
> lo_tell(), etc.
> All these functions just needs to be called inside BEGIN block. And yes,
> it can be done asynchronously.

Thanks for reminding me. I totally forgot about them.

>> > Though I cannot guarantee anything, I may be interested into working out
>> > a patch, if no one is already doing the same (of course I understand
>> > that this patch wouldn't be for 9.3, which is already in its late
>> > release cycle).
>> >
>> > Do you think this may be of interest?
>>
>> Yes, I understand your pain, and I myself think we need new APIs for
>> large objects. Probably that would be not terribly hard. One idea
>> would be inventing an asynchronous version of PQfn and let
>> lo_read/lo_write allow to use the new API.
>>
> Yes, but according to
> http://www.postgresql.org/docs/9.2/static/protocol-flow.html#AEN95330
> and/or http://www.postgresql.org/docs/9.2/static/libpq-fastpath.html
> function call sub-protocol is obsolete. Thats why personally I decided to
> use prepared statements.

I'm not totally pleased with the comment in the doc.  For me the only
reason why those extended protocol functions are recommended is the
binary protocol can be used. The price is parsing, planning, and
preparing the query, all of them are essentially unnecessary for a
large object access use case.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp