Thread: Large queries - again...
Hello all,
I've tried again sending large queries using libpq on Windows
environment, without success. I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using
Microsoft's Visual C++ 6.0, and tried sending a large query. The problem is, when the query is > 8192 large, a
NULLpointer is
returned from PQexec(). I have tried using ZDE (http://www.zeoslib.org), which I helped
develop, and pgAccess. ZDE is based on the Zeos Database Objects library,
which provides full access to PostgreSQL to Borland Delphi and Borland C++
Builder compilers. Could anyone please try this query:
ftp://carcass.dhs.org/pub/test.zip on windows (using libpq) and confirm it
suceed ? This archive contains a test.sql source, which will create a dumb
table with a text filed and then try to insert in it a large data (>8192
bytes) on it, and the libpq.dll I just compiled, for who want a fresh libpq
(it's virus free, don't worry... ). All my current PostgreSQL driver
implementation is depending on this. I'm sure the libpq will fail, unless
something very weird is happening in here... :) Other friends have confirmed this behaviour. I tried to
lookat the libpq sources to find out where's the error,
but I think it will take much less time to who develops it...
Best Regards,
Steve Howe
"Steve Howe" <howe@carcass.dhs.org> writes:
> I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using
> Microsoft's Visual C++ 6.0, and tried sending a large query.
> The problem is, when the query is > 8192 large, a NULL pointer is
> returned from PQexec().
It sure sounds to me like you are invoking an old (6.5 or before) libpq.
Perhaps you should check around to see if there are multiple libpq.dll
files on your system ...
regards, tom lane
Well, I tested the query you sent, and I got these results accessing the data: 1) libpq from Windows (freshly compiled from 7.1.2 sources): Error: pqReadData() -- read() failed: errno=0 No error 2) ODBC from Windows: It works ok. Steve Howe <howe@carcass.dhs.org> escreveu nas not�cias de mensagem:9j8jce$ddo$1@news.tht.net... > Hello all, > > I've tried again sending large queries using libpq on Windows > environment, without success. > I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll using > Microsoft's Visual C++ 6.0, and tried sending a large query. > The problem is, when the query is > 8192 large, a NULL pointer is > returned from PQexec(). > I have tried using ZDE (http://www.zeoslib.org), which I helped > develop, and pgAccess. ZDE is based on the Zeos Database Objects library, > which provides full access to PostgreSQL to Borland Delphi and Borland C++ > Builder compilers. > Could anyone please try this query: > ftp://carcass.dhs.org/pub/test.zip on windows (using libpq) and confirm it > suceed ? This archive contains a test.sql source, which will create a dumb > table with a text filed and then try to insert in it a large data (>8192 > bytes) on it, and the libpq.dll I just compiled, for who want a fresh libpq > (it's virus free, don't worry... ). All my current PostgreSQL driver > implementation is depending on this. I'm sure the libpq will fail, unless > something very weird is happening in here... :) > Other friends have confirmed this behaviour. > I tried to look at the libpq sources to find out where's the error, > but I think it will take much less time to who develops it... > > Best Regards, > Steve Howe > >
Hello Tom,
Nope, I'm 100% sure that the libpq.dll used is the one I just
compiled. And I never installed an older libpq.dll on this system. My application loads specifically the
libpq.dllI compiled (I use
the full library path on the call to LoadLibrary() call, so there is no way
it is an older library. Eduardo Stern from dbExperts (http://www.dbexperts.com.br), a
PostgreSQL specialized consulting company, got the same results. The ODBC driver, however, do not suffer from the
sameproblem, once
it does not use libpq. Maybe I should write a C sample that proves libpq under windows has
this bug ???
Best Regards,
Steve Howe
----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 12:35 PM
Subject: Re: [HACKERS] Large queries - again...
> "Steve Howe" <howe@carcass.dhs.org> writes:
> > I downloaded the PostgreSQL v7.12 sources, compiled libpq.dll
using
> > Microsoft's Visual C++ 6.0, and tried sending a large query.
> > The problem is, when the query is > 8192 large, a NULL pointer
is
> > returned from PQexec().
>
> It sure sounds to me like you are invoking an old (6.5 or before) libpq.
> Perhaps you should check around to see if there are multiple libpq.dll
> files on your system ...
>
> regards, tom lane
>
Hello Tom,
It returns "Error: pqReadData() -- read() failed: errno=0 No error
" as expected when a nil pointer is returned.
Best Regards,
Steve Howe
----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: <pgsql-hackers@postgresql.org>
Sent: Friday, July 20, 2001 2:42 PM
Subject: Re: [HACKERS] Large queries - again...
> "Steve Howe" <howe@carcass.dhs.org> writes:
> > Nope, I'm 100% sure that the libpq.dll used is the one I just
> > compiled. And I never installed an older libpq.dll on this system.
>
> Hmph. So what is left in PQerrorMessage() after the failure?
>
> regards, tom lane
>
"Steve Howe" <howe@carcass.dhs.org> writes:
> Nope, I'm 100% sure that the libpq.dll used is the one I just
> compiled. And I never installed an older libpq.dll on this system.
Hmph. So what is left in PQerrorMessage() after the failure?
regards, tom lane
"Steve Howe" <howe@carcass.dhs.org> writes:
> It returns "Error: pqReadData() -- read() failed: errno=0 No error
> " as expected when a nil pointer is returned.
"As expected"? That's not what I'd expect, especially not for a
behavior that's dependent on the size of an *outgoing* message.
(Thinks for awhile...) You're not using PQsetnonblocking() are you,
by any chance?
regards, tom lane
----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Steve Howe" <howe@carcass.dhs.org> Cc: <pgsql-hackers@postgresql.org> Sent: Friday, July 20, 2001 5:17 PM Subject: Re: [HACKERS] Large queries - again... > "Steve Howe" <howe@carcass.dhs.org> writes: > > It returns "Error: pqReadData() -- read() failed: errno=0 No error > > " as expected when a nil pointer is returned. > > "As expected"? That's not what I'd expect, especially not for a > behavior that's dependent on the size of an *outgoing* message. It is expected, because it's the default message when a PQexec() query returns NULL: pqReadData() will return nothing yet no error is signed. Of course, the "really expected" would be a sucessfull exec :-) > (Thinks for awhile...) You're not using PQsetnonblocking() are you, > by any chance? No, I'm not. Asynchronous libpq connections on Windows are still not realiable (althought I read someone submitted a patch recently), so I'm keeping synchronous queries for a while. I'm not also using any non-standard functions; just plain PQconnectdb() and PQexec()... Best Regards, Steve
> > "As expected"? That's not what I'd expect, especially not for a > > behavior that's dependent on the size of an *outgoing* message. > It is expected, because it's the default message when a PQexec() query > returns NULL: pqReadData() will return nothing yet no error is signed. > Of course, the "really expected" would be a sucessfull exec :-) > > > (Thinks for awhile...) You're not using PQsetnonblocking() are you, > > by any chance? > No, I'm not. Asynchronous libpq connections on Windows are still not > realiable (althought I read someone submitted a patch recently), so I'm > keeping synchronous queries for a while. I'm not also using any non-standard > functions; just plain PQconnectdb() and PQexec()... Yes, just applied. I will have another one next week. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
"Steve Howe" <howe@carcass.dhs.org> writes:
>> (Thinks for awhile...) You're not using PQsetnonblocking() are you,
>> by any chance?
> No, I'm not.
Drat, another perfectly good theory down the drain :-(.
Well, we're not going to find out anymore until we discover what the
error code actually is --- the "errno=0" bogosity isn't helping.
As Bruce mentioned, we did just commit a patch that #defines errno
as WSAGetLastError() on WIN32, so that you can get at least something
useful about socket errors. I'd suggest pulling the current CVS sources
(or a nightly snapshot tarball dated after today) and building libpq
from that. Then maybe we can learn more.
regards, tom lane
> "Steve Howe" <howe@carcass.dhs.org> writes:
> >> (Thinks for awhile...) You're not using PQsetnonblocking() are you,
> >> by any chance?
>
> > No, I'm not.
>
> Drat, another perfectly good theory down the drain :-(.
>
> Well, we're not going to find out anymore until we discover what the
> error code actually is --- the "errno=0" bogosity isn't helping.
> As Bruce mentioned, we did just commit a patch that #defines errno
> as WSAGetLastError() on WIN32, so that you can get at least something
> useful about socket errors. I'd suggest pulling the current CVS sources
> (or a nightly snapshot tarball dated after today) and building libpq
> from that. Then maybe we can learn more.
Unhappyly, there are unresolved externals and it won't build...
I'll try to fix it.
The log follows right below...
Best regards,
Steve Howe
----------------------------------------------------------------------------
----------
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
cd include if not exist config.h copy config.h.win32 config.h cd .. cd interfaces\libpq
nmake/f win32.mak
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588. Creating library .\Release\libpqdll.lib and object
.\Release\libpqdll.exp
libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf
libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog
.\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals
NMAKE : fatal error U1077: 'link.exe' : return code '0x460'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual
Studio\VC98\bin\NMAKE.EXE"' : return
code '0x2'
Stop.
OK, I just applied a patch to add the final fixes to Win32 libpq. Please try the CVS or later snapshot to see how it works. The patch suggested adding #define snprintf _snprintf to win32.h and I have done that. There was already one there for vsnprintf. I am quite confused about the elog() mention. I don't see where we added a call to elog() in the past day. I only see two mentions of elog in the code, both it dllist.c. They don't use elog() if you define FRONTEND. Please do -DFRONTEND on the compile line. I think this will give you a good library binary. Let us know how the new code works. The most recent patch I just applied was tested by a user and it worked well for him. Nice to have this resolved. I can mark this TODO item as done: * -Fix libpq to properly handle socket failures under native MS Win32 [libpq] > > "Steve Howe" <howe@carcass.dhs.org> writes: > > >> (Thinks for awhile...) You're not using PQsetnonblocking() are you, > > >> by any chance? > > > > > No, I'm not. > > > > Drat, another perfectly good theory down the drain :-(. > > > > Well, we're not going to find out anymore until we discover what the > > error code actually is --- the "errno=0" bogosity isn't helping. > > As Bruce mentioned, we did just commit a patch that #defines errno > > as WSAGetLastError() on WIN32, so that you can get at least something > > useful about socket errors. I'd suggest pulling the current CVS sources > > (or a nightly snapshot tarball dated after today) and building libpq > > from that. Then maybe we can learn more. > Unhappyly, there are unresolved externals and it won't build... > I'll try to fix it. > The log follows right below... > > Best regards, > Steve Howe > > ---------------------------------------------------------------------------- > ---------- > Microsoft (R) Program Maintenance Utility Version 6.00.8168.0 > Copyright (C) Microsoft Corp 1988-1998. All rights reserved. > > cd include > if not exist config.h copy config.h.win32 config.h > cd .. > cd interfaces\libpq > nmake /f win32.mak > > Microsoft (R) Program Maintenance Utility Version 6.00.8168.0 > Copyright (C) Microsoft Corp 1988-1998. All rights reserved. > > link.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01588. > Creating library .\Release\libpqdll.lib and object .\Release\libpqdll.exp > libpq.lib(fe-exec.obj) : error LNK2001: unresolved external symbol _snprintf > libpq.lib(fe-misc.obj) : error LNK2001: unresolved external symbol _snprintf > libpq.lib(fe-auth.obj) : error LNK2001: unresolved external symbol _snprintf > libpq.lib(dllist.obj) : error LNK2001: unresolved external symbol _elog > .\Release\libpq.dll : fatal error LNK1120: 2 unresolved externals > NMAKE : fatal error U1077: 'link.exe' : return code '0x460' > Stop. > NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual > Studio\VC98\bin\NMAKE.EXE"' : return > code '0x2' > Stop. > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://www.postgresql.org/search.mpl > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
----- Original Message -----
From: "Bruce Momjian" <pgman@candle.pha.pa.us>
To: "Steve Howe" <howe@carcass.dhs.org>
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>; <pgsql-hackers@postgresql.org>
Sent: Saturday, July 21, 2001 1:39 AM
Subject: Re: [HACKERS] Large queries - again...
>
> OK, I just applied a patch to add the final fixes to Win32 libpq.
> Please try the CVS or later snapshot to see how it works. The patch
> suggested adding
>
> #define snprintf _snprintf
>
> to win32.h and I have done that. There was already one there for
> vsnprintf. I am quite confused about the elog() mention. I don't see
> where we added a call to elog() in the past day. I only see two
> mentions of elog in the code, both it dllist.c. They don't use elog()
> if you define FRONTEND. Please do -DFRONTEND on the compile line. I
> think this will give you a good library binary.
I did it, but that brings other dependency problems (see below). I think
it's better to properly fix the elog issue... :-)
----------------------------------------------------------------------------
--------------------------------------------
C:\ttt\src>nmake -f win32.mak
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
cd include if not exist config.h copy config.h.win32 config.h cd .. cd interfaces\libpq
nmake/f win32.mak
Microsoft (R) Program Maintenance Utility Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
cl.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nma01700.
dllist.c
..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
file: 'sysexits.h': No such
file or directory
----------------------------------------------------------------------------
--------------------------------------------
> Let us know how the new code works. The most recent patch I just
> applied was tested by a user and it worked well for him. Nice to have
> this resolved. I can mark this TODO item as done:
>
> * -Fix libpq to properly handle socket failures under native MS
> Win32 [libpq]
I want this fixed more then anybody else i the world, believe me :-)
Best Regards,
Steve Howe
"Steve Howe" <howe@carcass.dhs.org> writes:
> ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include
> file: 'sysexits.h': No such file or directory
Jan added that recently. I was wondering if it was portable or not ...
looks like now we know :-(.
For the moment, just take out the include --- you may also need to
replace "exit(EX_UNAVAILABLE)" by plain "exit(1)".
regards, tom lane
> > OK, I just applied a patch to add the final fixes to Win32 libpq. > > Please try the CVS or later snapshot to see how it works. The patch > > suggested adding > > > > #define snprintf _snprintf > > > > to win32.h and I have done that. There was already one there for > > vsnprintf. I am quite confused about the elog() mention. I don't see > > where we added a call to elog() in the past day. I only see two > > mentions of elog in the code, both it dllist.c. They don't use elog() > > if you define FRONTEND. Please do -DFRONTEND on the compile line. I > > think this will give you a good library binary. > > I did it, but that brings other dependency problems (see below). I think > it's better to properly fix the elog issue... :-) Shouldn't we be defining FRONTEND in the win32.mak file? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Tom Lane wrote: > "Steve Howe" <howe@carcass.dhs.org> writes: > > ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include > > file: 'sysexits.h': No such file or directory > > Jan added that recently. I was wondering if it was portable or not ... > looks like now we know :-(. Grmbl - tell me why I don't like Windows... Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com # _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
> Tom Lane wrote: > > "Steve Howe" <howe@carcass.dhs.org> writes: > > > ..\..\backend\lib\dllist.c(20) : fatal error C1083: Cannot open include > > > file: 'sysexits.h': No such file or directory > > > > Jan added that recently. I was wondering if it was portable or not ... > > looks like now we know :-(. > > Grmbl - tell me why I don't like Windows... Please notify me when it's fixed so that I can test it. I'll also test the error messages returned from libpq on Windows, as requested on another thread. Best Regards, Steve Howe