Thread: Large queries - again...

Large queries - again...

From
"Steve Howe"
Date:
Hello all,
       I've tried again sending large queries using libpq on Windows
environment, without success.       I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using
Microsoft's Visual C++ 6.0, and tried sending a large query.       The problem is, when the query is > 8192 large, a
NULLpointer is
 
returned from PQexec().       I have tried using ZDE (http://www.zeoslib.org), which I helped
develop, and pgAccess. ZDE is based on the Zeos Database Objects library,
which provides full access to PostgreSQL to Borland Delphi and Borland C++
Builder compilers.       Could anyone please try this query:
ftp://carcass.dhs.org/pub/test.zip on windows (using libpq) and confirm it
suceed ? This archive contains a test.sql source, which will create a dumb
table with a text filed and then try to insert in it a large data (>8192
bytes) on it, and the libpq.dll I just compiled, for who want a fresh libpq
(it's virus free, don't worry... ). All my current PostgreSQL driver
implementation is depending on this. I'm sure the libpq will fail, unless
something very weird is happening in here... :)       Other friends have confirmed this behaviour.       I tried to
lookat the libpq sources to find out where's the error,
 
but I think it will take much less time to who develops it...

Best Regards,
Steve Howe




Re: Large queries - again...

From
Tom Lane
Date:
"Steve Howe" <howe@carcass.dhs.org> writes:
>         I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using
> Microsoft's Visual C++ 6.0, and tried sending a large query.
>         The problem is, when the query is > 8192 large, a NULL pointer is
> returned from PQexec().

It sure sounds to me like you are invoking an old (6.5 or before) libpq.
Perhaps you should check around to see if there are multiple libpq.dll
files on your system ...
        regards, tom lane


Re: Large queries - again...

From
"Eduardo Stern"
Date:
Well, I tested the query you sent, and I got these results accessing the
data:

1) libpq from Windows (freshly compiled from 7.1.2 sources): Error:
pqReadData() --  read() failed: errno=0
No error
2) ODBC from Windows: It works ok.




Steve Howe <howe@carcass.dhs.org> escreveu nas not�cias de
mensagem:9j8jce$ddo$1@news.tht.net...
> Hello all,
>
>         I've tried again sending large queries using libpq on Windows
> environment, without success.
>         I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll
using
> Microsoft's Visual C++ 6.0, and tried sending a large query.
>         The problem is, when the query is > 8192 large, a NULL pointer is
> returned from PQexec().
>         I have tried using ZDE (http://www.zeoslib.org), which I helped
> develop, and pgAccess. ZDE is based on the Zeos Database Objects library,
> which provides full access to PostgreSQL to Borland Delphi and Borland C++
> Builder compilers.
>         Could anyone please try this query:
> ftp://carcass.dhs.org/pub/test.zip on windows (using libpq) and confirm it
> suceed ? This archive contains a test.sql source, which will create a dumb
> table with a text filed and then try to insert in it a large data (>8192
> bytes) on it, and the libpq.dll I just compiled, for who want a fresh
libpq
> (it's virus free, don't worry... ). All my current PostgreSQL driver
> implementation is depending on this. I'm sure the libpq will fail, unless
> something very weird is happening in here... :)
>         Other friends have confirmed this behaviour.
>         I tried to look at the libpq sources to find out where's the
error,
> but I think it will take much less time to who develops it...
>
> Best Regards,
> Steve Howe
>
>




Re: Large queries - again...

From
"Steve Howe"
Date:
Hello Tom,
       Nope, I'm 100% sure that the libpq.dll used is the one I just
compiled. And I never installed an older libpq.dll on this system.       My application loads specifically the
libpq.dllI compiled (I use
 
the full library path on the call to LoadLibrary() call, so there is no way
it is an older library.       Eduardo Stern from dbExperts (http://www.dbexperts.com.br), a
PostgreSQL specialized consulting company, got the same results.       The ODBC driver, however, do not suffer from the
sameproblem, once
 
it does not use libpq.       Maybe I should write a C sample that proves libpq under windows has
this bug ???

Best Regards,
Steve Howe

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 12:35 PM
Subject: Re: [HACKERS] Large queries - again...


> "Steve Howe" <howe@carcass.dhs.org> writes:
> >         I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll
using
> > Microsoft's Visual C++ 6.0, and tried sending a large query.
> >         The problem is, when the query is > 8192 large, a NULL pointer
is
> > returned from PQexec().
>
> It sure sounds to me like you are invoking an old (6.5 or before) libpq.
> Perhaps you should check around to see if there are multiple libpq.dll
> files on your system ...
>
> regards, tom lane
>



Re: Large queries - again...

From
"Steve Howe"
Date:
Hello Tom,
       It returns "Error: pqReadData() --  read() failed: errno=0 No error
" as expected when a nil pointer is returned.

Best Regards,
Steve Howe

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 2:42 PM
Subject: Re: [HACKERS] Large queries - again...


> "Steve Howe" <howe@carcass.dhs.org> writes:
> >         Nope, I'm 100% sure that the libpq.dll used is the one I just
> > compiled. And I never installed an older libpq.dll on this system.
>
> Hmph.  So what is left in PQerrorMessage() after the failure?
>
> regards, tom lane
>



Re: Large queries - again...

From
Tom Lane
Date:
"Steve Howe" <howe@carcass.dhs.org> writes:
>         Nope, I'm 100% sure that the libpq.dll used is the one I just
> compiled. And I never installed an older libpq.dll on this system.

Hmph.  So what is left in PQerrorMessage() after the failure?
        regards, tom lane


Re: Large queries - again...

From
Tom Lane
Date:
"Steve Howe" <howe@carcass.dhs.org> writes:
>         It returns "Error: pqReadData() --  read() failed: errno=0 No error
> " as expected when a nil pointer is returned.

"As expected"?  That's not what I'd expect, especially not for a
behavior that's dependent on the size of an *outgoing* message.

(Thinks for awhile...)  You're not using PQsetnonblocking() are you,
by any chance?
        regards, tom lane


Re: Large queries - again...

From
"Steve Howe"
Date:
----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 5:17 PM
Subject: Re: [HACKERS] Large queries - again...


> "Steve Howe" <howe@carcass.dhs.org> writes:
> >         It returns "Error: pqReadData() --  read() failed: errno=0 No
error
> > " as expected when a nil pointer is returned.
>
> "As expected"?  That's not what I'd expect, especially not for a
> behavior that's dependent on the size of an *outgoing* message.
It is expected, because it's the default message when a PQexec() query
returns NULL: pqReadData() will return nothing yet no error is signed.
Of course, the "really expected" would be a sucessfull exec :-)

> (Thinks for awhile...)  You're not using PQsetnonblocking() are you,
> by any chance?
No, I'm not. Asynchronous libpq connections on Windows are still not
realiable (althought I read someone submitted a patch recently), so I'm
keeping synchronous queries for a while. I'm not also using any non-standard
functions; just plain PQconnectdb() and PQexec()...

Best Regards,
Steve



Re: Large queries - again...

From
Bruce Momjian
Date:
> > "As expected"?  That's not what I'd expect, especially not for a
> > behavior that's dependent on the size of an *outgoing* message.
> It is expected, because it's the default message when a PQexec() query
> returns NULL: pqReadData() will return nothing yet no error is signed.
> Of course, the "really expected" would be a sucessfull exec :-)
> 
> > (Thinks for awhile...)  You're not using PQsetnonblocking() are you,
> > by any chance?
> No, I'm not. Asynchronous libpq connections on Windows are still not
> realiable (althought I read someone submitted a patch recently), so I'm
> keeping synchronous queries for a while. I'm not also using any non-standard
> functions; just plain PQconnectdb() and PQexec()...

Yes, just applied.  I will have another one next week.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Large queries - again...

From
Tom Lane
Date:
"Steve Howe" <howe@carcass.dhs.org> writes:
>> (Thinks for awhile...)  You're not using PQsetnonblocking() are you,
>> by any chance?

> No, I'm not.

Drat, another perfectly good theory down the drain :-(.

Well, we're not going to find out anymore until we discover what the
error code actually is --- the "errno=0" bogosity isn't helping.
As Bruce mentioned, we did just commit a patch that #defines errno
as WSAGetLastError() on WIN32, so that you can get at least something
useful about socket errors.  I'd suggest pulling the current CVS sources
(or a nightly snapshot tarball dated after today) and building libpq
from that.  Then maybe we can learn more.
        regards, tom lane


Re: Large queries - again...

From
"Steve Howe"
Date:
> "Steve Howe" <howe@carcass.dhs.org> writes:
> >> (Thinks for awhile...)  You're not using PQsetnonblocking() are you,
> >> by any chance?
>
> > No, I'm not.
>
> Drat, another perfectly good theory down the drain :-(.
>
> Well, we're not going to find out anymore until we discover what the
> error code actually is --- the "errno=0" bogosity isn't helping.
> As Bruce mentioned, we did just commit a patch that #defines errno
> as WSAGetLastError() on WIN32, so that you can get at least something
> useful about socket errors.  I'd suggest pulling the current CVS sources
> (or a nightly snapshot tarball dated after today) and building libpq
> from that.  Then maybe we can learn more.
Unhappyly, there are unresolved externals and it won't build...
I'll try to fix it.
The log follows right below...

Best regards,
Steve Howe

----------------------------------------------------------------------------
----------
Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
       cd include       if not exist config.h copy config.h.win32 config.h       cd ..       cd interfaces\libpq
nmake/f win32.mak
 

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
       link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588.  Creating library .\Release\libpqdll.lib and object
.\Release\libpqdll.exp
libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog
.\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals
NMAKE : fatal error U1077: 'link.exe' : return code '0x460'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual
Studio\VC98\bin\NMAKE.EXE"' : return
code '0x2'
Stop.





Re: Large queries - again...

From
Bruce Momjian
Date:
OK, I just applied a patch to add the final fixes to Win32 libpq. 
Please try the CVS or later snapshot to see how it works.  The patch
suggested adding 
#define snprintf _snprintf

to win32.h and I have done that.  There was already one there for
vsnprintf.  I am quite confused about the elog() mention.  I don't see
where we added a call to elog() in the past day.  I only see two
mentions of elog in the code, both it dllist.c.  They don't use elog()
if you define FRONTEND.  Please do -DFRONTEND on the compile line.  I
think this will give you a good library binary.

Let us know how the new code works.  The most recent patch I just
applied was tested by a user and it worked well for him.  Nice to have
this resolved.  I can mark this TODO item as done:
* -Fix libpq to properly handle socket failures under native MS  Win32 [libpq]



> > "Steve Howe" <howe@carcass.dhs.org> writes:
> > >> (Thinks for awhile...)  You're not using PQsetnonblocking() are you,
> > >> by any chance?
> >
> > > No, I'm not.
> >
> > Drat, another perfectly good theory down the drain :-(.
> >
> > Well, we're not going to find out anymore until we discover what the
> > error code actually is --- the "errno=0" bogosity isn't helping.
> > As Bruce mentioned, we did just commit a patch that #defines errno
> > as WSAGetLastError() on WIN32, so that you can get at least something
> > useful about socket errors.  I'd suggest pulling the current CVS sources
> > (or a nightly snapshot tarball dated after today) and building libpq
> > from that.  Then maybe we can learn more.
> Unhappyly, there are unresolved externals and it won't build...
> I'll try to fix it.
> The log follows right below...
> 
> Best regards,
> Steve Howe
> 
> ----------------------------------------------------------------------------
> ----------
> Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
> Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
> 
>         cd include
>         if not exist config.h copy config.h.win32 config.h
>         cd ..
>         cd interfaces\libpq
>         nmake /f win32.mak
> 
> Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
> Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
> 
>         link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588.
>    Creating library .\Release\libpqdll.lib and object .\Release\libpqdll.exp
> libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf
> libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf
> libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf
> libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog
> .\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals
> NMAKE : fatal error U1077: 'link.exe' : return code '0x460'
> Stop.
> NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual
> Studio\VC98\bin\NMAKE.EXE"' : return
> code '0x2'
> Stop.
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://www.postgresql.org/search.mpl
> 

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Large queries - again...

From
"Steve Howe"
Date:
----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>; <pgsql-hackers@postgresql.org>
Sent: Saturday, July 21, 2001 1:39 AM
Subject: Re: [HACKERS] Large queries - again...


>
> OK, I just applied a patch to add the final fixes to Win32 libpq.
> Please try the CVS or later snapshot to see how it works.  The patch
> suggested adding
>
> #define snprintf _snprintf
>
> to win32.h and I have done that.  There was already one there for
> vsnprintf.  I am quite confused about the elog() mention.  I don't see
> where we added a call to elog() in the past day.  I only see two
> mentions of elog in the code, both it dllist.c.  They don't use elog()
> if you define FRONTEND.  Please do -DFRONTEND on the compile line.  I
> think this will give you a good library binary.

I did it, but that brings other dependency problems (see below). I think
it's better to properly fix the elog issue... :-)
----------------------------------------------------------------------------
--------------------------------------------
C:\ttt\src>nmake -f win32.mak

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
       cd include       if not exist config.h copy config.h.win32 config.h       cd ..       cd interfaces\libpq
nmake/f win32.mak
 

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
       cl.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01700.
dllist.c
..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
file: 'sysexits.h': No such
file or directory
----------------------------------------------------------------------------
--------------------------------------------
> Let us know how the new code works.  The most recent patch I just
> applied was tested by a user and it worked well for him.  Nice to have
> this resolved.  I can mark this TODO item as done:
>
> * -Fix libpq to properly handle socket failures under native MS
>   Win32 [libpq]
I want this fixed more then anybody else i the world, believe me  :-)

Best Regards,
Steve Howe



Re: Large queries - again...

From
Tom Lane
Date:
"Steve Howe" <howe@carcass.dhs.org> writes:
> ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
> file: 'sysexits.h': No such file or directory

Jan added that recently.  I was wondering if it was portable or not ...
looks like now we know :-(.

For the moment, just take out the include --- you may also need to
replace "exit(EX_UNAVAILABLE)" by plain "exit(1)".
        regards, tom lane


Re: Large queries - again...

From
Bruce Momjian
Date:
> > OK, I just applied a patch to add the final fixes to Win32 libpq.
> > Please try the CVS or later snapshot to see how it works.  The patch
> > suggested adding
> >
> > #define snprintf _snprintf
> >
> > to win32.h and I have done that.  There was already one there for
> > vsnprintf.  I am quite confused about the elog() mention.  I don't see
> > where we added a call to elog() in the past day.  I only see two
> > mentions of elog in the code, both it dllist.c.  They don't use elog()
> > if you define FRONTEND.  Please do -DFRONTEND on the compile line.  I
> > think this will give you a good library binary.
> 
> I did it, but that brings other dependency problems (see below). I think
> it's better to properly fix the elog issue... :-)

Shouldn't we be defining FRONTEND in the win32.mak file?

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Large queries - again...

From
Jan Wieck
Date:
Tom Lane wrote:
> "Steve Howe" <howe@carcass.dhs.org> writes:
> > ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
> > file: 'sysexits.h': No such file or directory
>
> Jan added that recently.  I was wondering if it was portable or not ...
> looks like now we know :-(.
   Grmbl - tell me why I don't like Windows...


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com



Re: Large queries - again...

From
"Steve Howe"
Date:
> Tom Lane wrote:
> > "Steve Howe" <howe@carcass.dhs.org> writes:
> > > ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open
include
> > > file: 'sysexits.h': No such file or directory
> >
> > Jan added that recently.  I was wondering if it was portable or not ...
> > looks like now we know :-(.
>
>     Grmbl - tell me why I don't like Windows...
Please notify me when it's fixed so that I can test it.
I'll also test the error messages returned from libpq on Windows, as
requested on  another thread.

Best Regards,
Steve Howe