Thread: Weird effects using BLOBs from libpq

Hi!

I'm experiencing somewhat confusing results using lo_creat(), lo_open(),
lo_write()...

First effect I have noticed is that when I write more than a certain amount
of data
at once using lo_write(), my application hangs indefinitely (10K work
perfect, 100K not).
Is there a well-known limit? On what does it depend? Or is it an unexpected
feature ;-)

Next off, now that everything runs fine, it does so only when using a local
database - when
connecting over the network to our test server, i need to put in usleeps of
minimum 30ms
in between consecutive writes to the same BLOB - else writing fails and i
have to close
the DB connection to do anything useful with the DB again!

Now, that's what i call unexpected...

I'm afraid i can't give you source code right now, 'cause i'm using LabVIEW
(a graphical
programming language).

Oh, yes, the local DB is postgreSQL 7.0.2 on cygwin 1.1.4, cygipc 1.0.7.
The remote is
7.0.2 on RedHat 6.2, single CPU.

Any ideas what could cause these effects? Are they by design (shudder)?

Greetings,
    Joerg

Joerg Hessdoerfer <Joerg.Hessdoerfer@sea-gmbh.com> writes:
> I'm experiencing somewhat confusing results using lo_creat(), lo_open(),
> lo_write()...

> First effect I have noticed is that when I write more than a certain
> amount of data at once using lo_write(), my application hangs
> indefinitely (10K work perfect, 100K not).  Is there a well-known
> limit? On what does it depend? Or is it an unexpected feature ;-)

> Next off, now that everything runs fine, it does so only when using a
> local database - when connecting over the network to our test server,
> i need to put in usleeps of minimum 30ms in between consecutive writes
> to the same BLOB - else writing fails and i have to close the DB
> connection to do anything useful with the DB again!

Ugh.  FWIW, no one else has reported anything like that, as far as I
can remember.

> I'm afraid i can't give you source code right now, 'cause i'm using LabVIEW
> (a graphical programming language).

> Oh, yes, the local DB is postgreSQL 7.0.2 on cygwin 1.1.4, cygipc 1.0.7.

Maybe it's just my natural dislike for anything Microsoft, but I'm
suspicious that you are looking at bugs in the Windows TCP/IP stack.
Or possibly a cygipc bug.  The timing dependency would more or less
have to be a TCP-level bug, seems like (unless you are trying to use
the "non blocking output" feature of libpq?  That's known not to be
too reliable, and I don't recommend using it...)

            regards, tom lane

At 17:52 22.10.00 -0400, you wrote:
[...]
> > First effect I have noticed is that when I write more than a certain
> > amount of data at once using lo_write(), my application hangs
> > indefinitely (10K work perfect, 100K not).  Is there a well-known
> > limit? On what does it depend? Or is it an unexpected feature ;-)
>
> > Next off, now that everything runs fine, it does so only when using a
> > local database - when connecting over the network to our test server,
> > i need to put in usleeps of minimum 30ms in between consecutive writes
> > to the same BLOB - else writing fails and i have to close the DB
> > connection to do anything useful with the DB again!
>
>Ugh.  FWIW, no one else has reported anything like that, as far as I
>can remember.

Wow - that's interesting. At least the transfersize issue seems unlikely to be
a WIN artefact.

> > I'm afraid i can't give you source code right now, 'cause i'm using
> LabVIEW
> > (a graphical programming language).
>
> > Oh, yes, the local DB is postgreSQL 7.0.2 on cygwin 1.1.4, cygipc 1.0.7.
>
>Maybe it's just my natural dislike for anything Microsoft, but I'm
>suspicious that you are looking at bugs in the Windows TCP/IP stack.

I'd second that - but i can verify that on Linux, given a few moments spare
time.
Perhaps later today.

>Or possibly a cygipc bug.  The timing dependency would more or less
>have to be a TCP-level bug, seems like (unless you are trying to use
>the "non blocking output" feature of libpq?  That's known not to be
>too reliable, and I don't recommend using it...)

No, i dont use non-blocking mode.

>                         regards, tom lane

Well, i'll do some tests on Linux then and post the results.

Thanx,
         Joerg

Hi!

Just as a follow-up to my last posting - disregard my complaints ;-)

I've just checked under Linux - works fine. Then i went back to my WIN NT box,
and double checked libraries. See what i found: i actually used a WIN32 build
of the libpq - not the cygwin build. When using that, everything is fine, too.

As cygwin uses WIN NT's IP stack, that indicates a poor port of libpq to
WIN32,
doesn't it? Question is, should we bother? One can always just resort to use
the cygwin stuff (you need only the cygwin1.dll, IMHO).

Perhaps i'll dig myself into the WIN32 code.

Greetings,
    Joerg

Joerg Hessdoerfer <Joerg.Hessdoerfer@sea-gmbh.com> writes:
> I've just checked under Linux - works fine. Then i went back to my WIN
> NT box, and double checked libraries. See what i found: i actually
> used a WIN32 build of the libpq - not the cygwin build. When using
> that, everything is fine, too.

OK, thanks for the report.

> As cygwin uses WIN NT's IP stack, that indicates a poor port of libpq
> to WIN32, doesn't it?

No, it means that Microsoft's implementation of the standard socket
calls is broken.  There is no way that user-level code should cause
a TCP implementation to drop data in the way you are seeing.

I was originally concerned that libpq itself was dropping data; the
"nonblock mode" code will do that, and you have to be very careful
to pay attention to whether data was actually sent or not.  (It's
fundamentally broken because the library itself fails to pay attention
in many places, but I digress.)  Since you're not using nonblock mode,
the library is just shoving out bytes with send(), and it's not our
responsibility to throttle the data flow; it's the TCP stack's.

It would seem that the problem is in the API layer that provides
Berkeley socket emulation, not in the underlying stack, but it's
still Microsoft's code.

> Question is, should we bother?

I can't get excited about it, but if you have the time and interest,
please do dig into it.

            regards, tom lane