Thread: libpq sockets on win32

libpq sockets on win32

From
"Jeff Johnson"
Date:
I found that sending more than 16k of data in a SQL statement caused a
problem.  I was using PyGreSQL but narrowed the error down to
"\postgresql-7.1.2\src\interfaces\libpq\fe-misc.c" pqReadData() and
pqFlush().  After some poking around, it looked like recv was setting
errno to ENOENT for some reason.  I figured out why today.  Win32
sockets don't set errno at all.  ENOENT was just left in errno from
some earlier call.

This article describes the problem and work around.
http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm

I haven't done much C coding in a few years and don't want to break
other code by blindly doing:

#define errno WSAGetLastError

Does anyone that knows this code want to take a crack at it?

Thanks,
Jeff


Re: libpq sockets on win32

From
Bruce Momjian
Date:
> I found that sending more than 16k of data in a SQL statement caused a
> problem.  I was using PyGreSQL but narrowed the error down to
> "\postgresql-7.1.2\src\interfaces\libpq\fe-misc.c" pqReadData() and
> pqFlush().  After some poking around, it looked like recv was setting
> errno to ENOENT for some reason.  I figured out why today.  Win32
> sockets don't set errno at all.  ENOENT was just left in errno from
> some earlier call.
>
> This article describes the problem and work around.
> http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm

I can't read that web site under Netscape.

> I haven't done much C coding in a few years and don't want to break
> other code by blindly doing:
>
> #define errno WSAGetLastError

If I could read it, I think I could fix it.  Please send it in some
readable format.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: libpq sockets on win32

From
Bruce Momjian
Date:
Can you try this patch and let me know if it helps?  It is a different
approach.  This was the only place I saw errno checked for a
non-predefined value.

One other solution may be to use the #define only in the libpq C files
that need it.  What really stinks is that the errno define is only
useful for socket errno settings.

I do see a use in fe-connect.c:

    #ifndef WIN32
            if (errno == EINPROGRESS || errno == 0)
    #else
            if (WSAGetLastError() == WSAEINPROGRESS)
    #endif

I hate to litter this through the whole source.  I wonder if we have to
bracket the errno checkes with #define/#undef.  Can you try that with
the fix described on the web page.  The above would convert to:

    #ifdef WIN32
    #define errno WSAGetLastError
    #endif
            if (errno == EINPROGRESS || errno == 0)
    #ifdef WIN32
    #undef errno
    #endif

Maybe make these into their own macros somehow.

> > > This article describes the problem and work around.
> > > http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm
> >
> > I can't read that web site under Netscape.
> >
> > If I could read it, I think I could fix it.  Please send it in some
> > readable format.
> >
>
> I chopped the content out and stuck it into a basic HTML file and
> attached it.
>
> Thanks for taking a look,
> Jeff

[ Attachment, skipping... ]

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
Index: src/interfaces/libpq/fe-misc.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/fe-misc.c,v
retrieving revision 1.49
diff -c -r1.49 fe-misc.c
*** src/interfaces/libpq/fe-misc.c    2001/05/28 15:29:51    1.49
--- src/interfaces/libpq/fe-misc.c    2001/06/04 17:52:40
***************
*** 614,619 ****
--- 614,623 ----

          int            sent;

+ #ifdef WIN32
+         errno = 0;    /* Win32 doesn't set this, needs reset */
+ #endif
+
  #ifdef USE_SSL
          if (conn->ssl)
              sent = SSL_write(conn->ssl, ptr, len);

RE: libpq sockets on win32

From
"Jeff Johnson"
Date:
> > This article describes the problem and work around.
> > http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm
>
> I can't read that web site under Netscape.
>
> If I could read it, I think I could fix it.  Please send it in some
> readable format.
>

I chopped the content out and stuck it into a basic HTML file and
attached it.

Thanks for taking a look,
Jeff

Re: libpq sockets on win32

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> This article describes the problem and work around.
>> http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm

> I can't read that web site under Netscape.

Comes up fine for me (maybe you need to turn off Javascript?)

However, the advice on it seems to be written by someone who does not
know the difference between a variable and a function :-(.  Read with
a large grain of salt.

We already have a couple of #ifdef'd usages of WSAGetLastError() in
libpq, but it's quite unclear which other references to errno might
need to be changed.  A blanket replacement of errno would certainly
break things.

            regards, tom lane

Re: libpq sockets on win32

From
Peter Eisentraut
Date:
Jeff Johnson writes:

> After some poking around, it looked like recv was setting errno to
> ENOENT for some reason.  I figured out why today.  Win32 sockets don't
> set errno at all.  ENOENT was just left in errno from some earlier
> call.

There are already a few (wrong) attempts to cover this situation in
fe-connect.c, but it looks like a bunch of #ifdef's are needed.

> I haven't done much C coding in a few years and don't want to break
> other code by blindly doing:
>
> #define errno WSAGetLastError

Probably rather

#ifndef WIN32
normal code;
#else
windows code;
#endif

-- 
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter



Re: libpq sockets on win32

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> This article describes the problem and work around.
> >> http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm
>
> > I can't read that web site under Netscape.
>
> Comes up fine for me (maybe you need to turn off Javascript?)

Thanks, that fixed it.

> However, the advice on it seems to be written by someone who does not
> know the difference between a variable and a function :-(.  Read with
> a large grain of salt.
>
> We already have a couple of #ifdef'd usages of WSAGetLastError() in
> libpq, but it's quite unclear which other references to errno might
> need to be changed.  A blanket replacement of errno would certainly
> break things.

That's why I recommended brackets of #define/#undef around tests of
socket errno's.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

RE: libpq sockets on win32

From
"Jeff Johnson"
Date:
Bruce Momjian wrote:
> Can you try this patch and let me know if it helps?  It is a
different
> approach.  This was the only place I saw errno checked for a
> non-predefined value.

Setting errno = 0 doesn't help, the error handling code is entered
when recv returns -1, then even if errno == 0, it'll bomb out.

> I hate to litter this through the whole source.  I wonder if
> we have to
> bracket the errno checkes with #define/#undef.  Can you try that
with
> the fix described on the web page.  The above would convert to:
>
>     #ifdef WIN32
>     #define errno WSAGetLastError
>     #endif
>             if (errno == EINPROGRESS || errno == 0)
>     #ifdef WIN32
>     #undef errno
>     #endif
>
> Maybe make these into their own macros somehow.

Even when I was a C programmer I never did much more than simple
defines with the pre-compiler so I'll leave this to those that know
how :)

As Tom Lane points out in another post, the "define errno
WSAGetLastError" seems to confuse a variable with a function.  I was
surprised that such a thing could work.  I'm happy to hear that it
doesn't.

What about something like this:

#ifdef WIN32
#define s_errno WSAGetLastError()
#else
#define s_errno errno
#endif

/* for socket functions, check s_errno */
if (s_errno == EINPROGRESS || s_errno == 0)
...

/* for non-socket functions, check errno as usual */
if (errno == ENOENT)
...


Then replace only errno with s_errno when it is used with socket code.
I'm not sure if strerror would work with all the errors returned by
WSAGetLastError().  The Win32 SDK says to use FormatMessage(a ton of
stuff here).


Regards,
Jeff


Re: libpq sockets on win32

From
Bruce Momjian
Date:
> As Tom Lane points out in another post, the "define errno
> WSAGetLastError" seems to confuse a variable with a function.  I was
> surprised that such a thing could work.  I'm happy to hear that it
> doesn't.
>
> What about something like this:
>
> #ifdef WIN32
> #define s_errno WSAGetLastError()
> #else
> #define s_errno errno
> #endif
>
> /* for socket functions, check s_errno */
> if (s_errno == EINPROGRESS || s_errno == 0)
> ...
>
> /* for non-socket functions, check errno as usual */
> if (errno == ENOENT)
> ...

I have done exactly that.  I assume fcntl(), ioctl(), select() use errno
even if used on a socket, while getsockopt(), setsockopt(), socket(),
connect(), getsockname(), send(), recv() use WSAGetLastError.  Is this
list correct?

The patch is attached.  Please let me know so I can finalize it and
apply it.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
Index: src/interfaces/libpq/fe-connect.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/fe-connect.c,v
retrieving revision 1.164
diff -c -r1.164 fe-connect.c
*** src/interfaces/libpq/fe-connect.c    2001/03/31 23:14:37    1.164
--- src/interfaces/libpq/fe-connect.c    2001/06/04 19:18:14
***************
*** 735,741 ****
      {
          printfPQExpBuffer(&conn->errorMessage,
                   "connectNoDelay() -- setsockopt failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
  #ifdef WIN32
          printf("Winsock error: %i\n", WSAGetLastError());
  #endif
--- 735,741 ----
      {
          printfPQExpBuffer(&conn->errorMessage,
                   "connectNoDelay() -- setsockopt failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
  #ifdef WIN32
          printf("Winsock error: %i\n", WSAGetLastError());
  #endif
***************
*** 890,896 ****
          printfPQExpBuffer(&conn->errorMessage,
                            "connectDBStart() -- "
                            "socket() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          goto connect_errReturn;
      }

--- 890,896 ----
          printfPQExpBuffer(&conn->errorMessage,
                            "connectDBStart() -- "
                            "socket() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          goto connect_errReturn;
      }

***************
*** 934,944 ****
       */
      if (connect(conn->sock, &conn->raddr.sa, conn->raddr_len) < 0)
      {
! #ifndef WIN32
!         if (errno == EINPROGRESS || errno == 0)
! #else
!         if (WSAGetLastError() == WSAEINPROGRESS)
! #endif
          {

              /*
--- 934,940 ----
       */
      if (connect(conn->sock, &conn->raddr.sa, conn->raddr_len) < 0)
      {
!         if (sockerrno == EINPROGRESS || sockerrno == 0)
          {

              /*
***************
*** 950,956 ****
          else
          {
              /* Something's gone wrong */
!             connectFailureMessage(conn, "connectDBStart()", errno);
              goto connect_errReturn;
          }
      }
--- 946,952 ----
          else
          {
              /* Something's gone wrong */
!             connectFailureMessage(conn, "connectDBStart()", sockerrno);
              goto connect_errReturn;
          }
      }
***************
*** 970,983 ****
          {
              printfPQExpBuffer(&conn->errorMessage,
                                "connectDB() -- couldn't send SSL negotiation packet: errno=%d\n%s\n",
!                               errno, strerror(errno));
              goto connect_errReturn;
          }
          /* Now receive the postmasters response */
          if (recv(conn->sock, &SSLok, 1, 0) != 1)
          {
              printfPQExpBuffer(&conn->errorMessage, "PQconnectDB() -- couldn't read postmaster response:
errno=%d\n%s\n",
!                               errno, strerror(errno));
              goto connect_errReturn;
          }
          if (SSLok == 'S')
--- 966,979 ----
          {
              printfPQExpBuffer(&conn->errorMessage,
                                "connectDB() -- couldn't send SSL negotiation packet: errno=%d\n%s\n",
!                               sockerrno, strerror(sockerrno));
              goto connect_errReturn;
          }
          /* Now receive the postmasters response */
          if (recv(conn->sock, &SSLok, 1, 0) != 1)
          {
              printfPQExpBuffer(&conn->errorMessage, "PQconnectDB() -- couldn't read postmaster response:
errno=%d\n%s\n",
!                               sockerrno, strerror(sockerrno));
              goto connect_errReturn;
          }
          if (SSLok == 'S')
***************
*** 1233,1239 ****
                      printfPQExpBuffer(&conn->errorMessage,
                                 "PQconnectPoll() -- getsockopt() failed: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }
                  else if (optval != 0)
--- 1229,1235 ----
                      printfPQExpBuffer(&conn->errorMessage,
                                 "PQconnectPoll() -- getsockopt() failed: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }
                  else if (optval != 0)
***************
*** 1255,1261 ****
                      printfPQExpBuffer(&conn->errorMessage,
                                "PQconnectPoll() -- getsockname() failed: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }

--- 1251,1257 ----
                      printfPQExpBuffer(&conn->errorMessage,
                                "PQconnectPoll() -- getsockname() failed: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }

***************
*** 1296,1302 ****
                                        "PQconnectPoll() --  "
                                        "couldn't send startup packet: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }

--- 1292,1298 ----
                                        "PQconnectPoll() --  "
                                        "couldn't send startup packet: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }

***************
*** 2110,2116 ****
--- 2106,2114 ----
  int
  PQrequestCancel(PGconn *conn)
  {
+ #ifndef WIN32
      int            save_errno = errno;
+ #endif
      int            tmpsock = -1;
      struct
      {
***************
*** 2127,2133 ****
--- 2125,2133 ----
          strcpy(conn->errorMessage.data,
                 "PQrequestCancel() -- connection is not open\n");
          conn->errorMessage.len = strlen(conn->errorMessage.data);
+ #ifndef WIN32
          errno = save_errno;
+ #endif
          return FALSE;
      }

***************
*** 2173,2183 ****
      close(tmpsock);
  #endif

      errno = save_errno;
      return TRUE;

  cancel_errReturn:
!     strcat(conn->errorMessage.data, strerror(errno));
      strcat(conn->errorMessage.data, "\n");
      conn->errorMessage.len = strlen(conn->errorMessage.data);
      if (tmpsock >= 0)
--- 2173,2185 ----
      close(tmpsock);
  #endif

+ #ifndef WIN32
      errno = save_errno;
+ #endif
      return TRUE;

  cancel_errReturn:
!     strcat(conn->errorMessage.data, strerror(sockerrno));
      strcat(conn->errorMessage.data, "\n");
      conn->errorMessage.len = strlen(conn->errorMessage.data);
      if (tmpsock >= 0)
***************
*** 2188,2194 ****
--- 2190,2198 ----
          close(tmpsock);
  #endif
      }
+ #ifndef WIN32
      errno = save_errno;
+ #endif
      return FALSE;
  }

Index: src/interfaces/libpq/fe-misc.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/fe-misc.c,v
retrieving revision 1.49
diff -c -r1.49 fe-misc.c
*** src/interfaces/libpq/fe-misc.c    2001/05/28 15:29:51    1.49
--- src/interfaces/libpq/fe-misc.c    2001/06/04 19:18:20
***************
*** 447,471 ****
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (errno == EINTR)
              goto tryAgain;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (errno == EAGAIN)
              return someread;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (errno == EWOULDBLOCK)
              return someread;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (errno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          return -1;
      }
      if (nread > 0)
--- 447,471 ----
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (sockerrno == EINTR)
              goto tryAgain;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (sockerrno == EAGAIN)
              return someread;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (sockerrno == EWOULDBLOCK)
              return someread;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (sockerrno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          return -1;
      }
      if (nread > 0)
***************
*** 533,557 ****
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (errno == EINTR)
              goto tryAgain2;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (errno == EAGAIN)
              return 0;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (errno == EWOULDBLOCK)
              return 0;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (errno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          return -1;
      }
      if (nread > 0)
--- 533,557 ----
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (sockerrno == EINTR)
              goto tryAgain2;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (sockerrno == EAGAIN)
              return 0;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (sockerrno == EWOULDBLOCK)
              return 0;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (sockerrno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          return -1;
      }
      if (nread > 0)
***************
*** 633,639 ****
               * EPIPE or ECONNRESET, assume we've lost the backend
               * connection permanently.
               */
!             switch (errno)
              {
  #ifdef EAGAIN
                  case EAGAIN:
--- 633,639 ----
               * EPIPE or ECONNRESET, assume we've lost the backend
               * connection permanently.
               */
!             switch (sockerrno)
              {
  #ifdef EAGAIN
                  case EAGAIN:
***************
*** 668,674 ****
                  default:
                      printfPQExpBuffer(&conn->errorMessage,
                        "pqFlush() --  couldn't send data: errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      /* We don't assume it's a fatal error... */
                      return EOF;
              }
--- 668,674 ----
                  default:
                      printfPQExpBuffer(&conn->errorMessage,
                        "pqFlush() --  couldn't send data: errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      /* We don't assume it's a fatal error... */
                      return EOF;
              }
Index: src/interfaces/libpq/libpq-int.h
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/libpq-int.h,v
retrieving revision 1.33
diff -c -r1.33 libpq-int.h
*** src/interfaces/libpq/libpq-int.h    2001/03/22 04:01:27    1.33
--- src/interfaces/libpq/libpq-int.h    2001/06/04 19:18:23
***************
*** 34,39 ****
--- 34,45 ----
  #include <openssl/err.h>
  #endif

+ #ifndef WIN32
+ #define sockerrno errno
+ #else
+ #define sockerrno WSAGetLastError
+ #endif
+
  /* libpq supports this version of the frontend/backend protocol.
   *
   * NB: we used to use PG_PROTOCOL_LATEST from the backend pqcomm.h file,

RE: libpq sockets on win32

From
"Jeff Johnson"
Date:
> I have done exactly that.  I assume fcntl(), ioctl(),
> select() use errno
> even if used on a socket, while getsockopt(), setsockopt(),
socket(),
> connect(), getsockname(), send(), recv() use WSAGetLastError.  Is
this
> list correct?

I don't know enough about such things.

>
> The patch is attached.  Please let me know so I can finalize it and
> apply it.

Couple of changes required to compile on Win32:

Change:
#define sockerrno WSAGetLastError
To:
#define sockerrno WSAGetLastError()


This has to go back into fe-connect.c, EINPROGRESS isn't defined in
Win32 for some reason..
#ifndef WIN32
        if (errno == EINPROGRESS || errno == 0)
#else
        if (WSAGetLastError() == WSAEINPROGRESS)
#endif



I tested it out but I'm getting this error when sending a large SQL
statement (>16k).


  File "WebKit\Application.py", line 340, in dispatchRequest
    self.handleGoodURL(transaction)
  File "WebKit\Application.py", line 457, in handleGoodURL
    self.respond(transaction)
  File "WebKit\Application.py", line 569, in respond
    transaction.respond()
  File "H:\webware\Webware\WebKit\Transaction.py", line 93, in respond
    self._servlet.respond(self)
  File "H:\webware\Webware\WebKit\HTTPServlet.py", line 38, in respond
    method(self, trans)
  File "H:\webware\Webware\WebKit\Page.py", line 34, in respondToGet
    self._respond(transaction)
  File "H:\webware\Webware\WebKit\Page.py", line 67, in _respond
    self.writeHTML()
  File "H:\webware\Webware\WebKit\Page.py", line 129, in writeHTML
    self.writeBody()
  File "H:\webware\Webware\WebKit\Page.py", line 186, in writeBody
    self.writeBodyParts()
  File "/EA\www\SitePage.py", line 305, in writeBodyParts
    self.writeContent()
  File "\EA\www\MyTest.py", line 9, in writeContent
    self.transferRecords()
  File "\EA\www\MyTest.py", line 45, in transferRecords
    cu.execute(sql)
  File "c:\python20\pgdb.py", line 189, in execute
    self.executemany(operation, (params,))
  File "c:\python20\pgdb.py", line 204, in executemany
    rows = self.__source.execute(sql)
ValueError: pqReadData() --  read() failed: errno=10035
Unknown error




Re: libpq sockets on win32

From
Bruce Momjian
Date:
> > I have done exactly that.  I assume fcntl(), ioctl(),
> > select() use errno
> > even if used on a socket, while getsockopt(), setsockopt(),
> socket(),
> > connect(), getsockname(), send(), recv() use WSAGetLastError.  Is
> this
> > list correct?
>
> I don't know enough about such things.

The web page wasn't clear about that.

> >
> > The patch is attached.  Please let me know so I can finalize it and
> > apply it.
>
> Couple of changes required to compile on Win32:
>
> Change:
> #define sockerrno WSAGetLastError
> To:
> #define sockerrno WSAGetLastError()
>

OK.

>
> This has to go back into fe-connect.c, EINPROGRESS isn't defined in
> Win32 for some reason..
> #ifndef WIN32
>         if (errno == EINPROGRESS || errno == 0)
> #else
>         if (WSAGetLastError() == WSAEINPROGRESS)
> #endif
>

OK.  Not sure why it wasn't defined, but, oh well.

>
>
> I tested it out but I'm getting this error when sending a large SQL
> statement (>16k).
>     cu.execute(sql)
>   File "c:\python20\pgdb.py", line 189, in execute
>     self.executemany(operation, (params,))
>   File "c:\python20\pgdb.py", line 204, in executemany
>     rows = self.__source.execute(sql)
> ValueError: pqReadData() --  read() failed: errno=10035
> Unknown error

That errno is very high, 10035.  If I take a look at my sys/errno.h on
BSD/OS, I see:

    #define EAGAIN      35      /* Resource temporarily unavailable */

so my guess is that WSAGetLastError() returns errno plus 10,000.  What
value does your include file set for EAGAIN, 10035 or 35.  If it is 35,
the following patch may work.  It has all your suggested fixes.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
Index: src/interfaces/libpq/fe-connect.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/fe-connect.c,v
retrieving revision 1.164
diff -c -r1.164 fe-connect.c
*** src/interfaces/libpq/fe-connect.c    2001/03/31 23:14:37    1.164
--- src/interfaces/libpq/fe-connect.c    2001/06/04 19:18:14
***************
*** 735,741 ****
      {
          printfPQExpBuffer(&conn->errorMessage,
                   "connectNoDelay() -- setsockopt failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
  #ifdef WIN32
          printf("Winsock error: %i\n", WSAGetLastError());
  #endif
--- 735,741 ----
      {
          printfPQExpBuffer(&conn->errorMessage,
                   "connectNoDelay() -- setsockopt failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
  #ifdef WIN32
          printf("Winsock error: %i\n", WSAGetLastError());
  #endif
***************
*** 890,896 ****
          printfPQExpBuffer(&conn->errorMessage,
                            "connectDBStart() -- "
                            "socket() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          goto connect_errReturn;
      }

--- 890,896 ----
          printfPQExpBuffer(&conn->errorMessage,
                            "connectDBStart() -- "
                            "socket() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          goto connect_errReturn;
      }

***************
*** 950,956 ****
          else
          {
              /* Something's gone wrong */
!             connectFailureMessage(conn, "connectDBStart()", errno);
              goto connect_errReturn;
          }
      }
--- 946,952 ----
          else
          {
              /* Something's gone wrong */
!             connectFailureMessage(conn, "connectDBStart()", sockerrno);
              goto connect_errReturn;
          }
      }
***************
*** 970,983 ****
          {
              printfPQExpBuffer(&conn->errorMessage,
                                "connectDB() -- couldn't send SSL negotiation packet: errno=%d\n%s\n",
!                               errno, strerror(errno));
              goto connect_errReturn;
          }
          /* Now receive the postmasters response */
          if (recv(conn->sock, &SSLok, 1, 0) != 1)
          {
              printfPQExpBuffer(&conn->errorMessage, "PQconnectDB() -- couldn't read postmaster response:
errno=%d\n%s\n",
!                               errno, strerror(errno));
              goto connect_errReturn;
          }
          if (SSLok == 'S')
--- 966,979 ----
          {
              printfPQExpBuffer(&conn->errorMessage,
                                "connectDB() -- couldn't send SSL negotiation packet: errno=%d\n%s\n",
!                               sockerrno, strerror(sockerrno));
              goto connect_errReturn;
          }
          /* Now receive the postmasters response */
          if (recv(conn->sock, &SSLok, 1, 0) != 1)
          {
              printfPQExpBuffer(&conn->errorMessage, "PQconnectDB() -- couldn't read postmaster response:
errno=%d\n%s\n",
!                               sockerrno, strerror(sockerrno));
              goto connect_errReturn;
          }
          if (SSLok == 'S')
***************
*** 1233,1239 ****
                      printfPQExpBuffer(&conn->errorMessage,
                                 "PQconnectPoll() -- getsockopt() failed: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }
                  else if (optval != 0)
--- 1229,1235 ----
                      printfPQExpBuffer(&conn->errorMessage,
                                 "PQconnectPoll() -- getsockopt() failed: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }
                  else if (optval != 0)
***************
*** 1255,1261 ****
                      printfPQExpBuffer(&conn->errorMessage,
                                "PQconnectPoll() -- getsockname() failed: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }

--- 1251,1257 ----
                      printfPQExpBuffer(&conn->errorMessage,
                                "PQconnectPoll() -- getsockname() failed: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }

***************
*** 1296,1302 ****
                                        "PQconnectPoll() --  "
                                        "couldn't send startup packet: "
                                        "errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      goto error_return;
                  }

--- 1292,1298 ----
                                        "PQconnectPoll() --  "
                                        "couldn't send startup packet: "
                                        "errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      goto error_return;
                  }

***************
*** 2110,2116 ****
--- 2106,2114 ----
  int
  PQrequestCancel(PGconn *conn)
  {
+ #ifndef WIN32
      int            save_errno = errno;
+ #endif
      int            tmpsock = -1;
      struct
      {
***************
*** 2127,2133 ****
--- 2125,2133 ----
          strcpy(conn->errorMessage.data,
                 "PQrequestCancel() -- connection is not open\n");
          conn->errorMessage.len = strlen(conn->errorMessage.data);
+ #ifndef WIN32
          errno = save_errno;
+ #endif
          return FALSE;
      }

***************
*** 2173,2183 ****
      close(tmpsock);
  #endif

      errno = save_errno;
      return TRUE;

  cancel_errReturn:
!     strcat(conn->errorMessage.data, strerror(errno));
      strcat(conn->errorMessage.data, "\n");
      conn->errorMessage.len = strlen(conn->errorMessage.data);
      if (tmpsock >= 0)
--- 2173,2185 ----
      close(tmpsock);
  #endif

+ #ifndef WIN32
      errno = save_errno;
+ #endif
      return TRUE;

  cancel_errReturn:
!     strcat(conn->errorMessage.data, strerror(sockerrno));
      strcat(conn->errorMessage.data, "\n");
      conn->errorMessage.len = strlen(conn->errorMessage.data);
      if (tmpsock >= 0)
***************
*** 2188,2194 ****
--- 2190,2198 ----
          close(tmpsock);
  #endif
      }
+ #ifndef WIN32
      errno = save_errno;
+ #endif
      return FALSE;
  }

Index: src/interfaces/libpq/fe-misc.c
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/fe-misc.c,v
retrieving revision 1.49
diff -c -r1.49 fe-misc.c
*** src/interfaces/libpq/fe-misc.c    2001/05/28 15:29:51    1.49
--- src/interfaces/libpq/fe-misc.c    2001/06/04 19:18:20
***************
*** 447,471 ****
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (errno == EINTR)
              goto tryAgain;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (errno == EAGAIN)
              return someread;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (errno == EWOULDBLOCK)
              return someread;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (errno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          return -1;
      }
      if (nread > 0)
--- 447,471 ----
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (sockerrno == EINTR)
              goto tryAgain;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (sockerrno == EAGAIN)
              return someread;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (sockerrno == EWOULDBLOCK)
              return someread;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (sockerrno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          return -1;
      }
      if (nread > 0)
***************
*** 533,557 ****
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (errno == EINTR)
              goto tryAgain2;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (errno == EAGAIN)
              return 0;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (errno == EWOULDBLOCK)
              return 0;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (errno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           errno, strerror(errno));
          return -1;
      }
      if (nread > 0)
--- 533,557 ----
                       conn->inBufSize - conn->inEnd, 0);
      if (nread < 0)
      {
!         if (sockerrno == EINTR)
              goto tryAgain2;
          /* Some systems return EAGAIN/EWOULDBLOCK for no data */
  #ifdef EAGAIN
!         if (sockerrno == EAGAIN)
              return 0;
  #endif
  #if defined(EWOULDBLOCK) && (!defined(EAGAIN) || (EWOULDBLOCK != EAGAIN))
!         if (sockerrno == EWOULDBLOCK)
              return 0;
  #endif
          /* We might get ECONNRESET here if using TCP and backend died */
  #ifdef ECONNRESET
!         if (sockerrno == ECONNRESET)
              goto definitelyFailed;
  #endif
          printfPQExpBuffer(&conn->errorMessage,
                          "pqReadData() --  read() failed: errno=%d\n%s\n",
!                           sockerrno, strerror(sockerrno));
          return -1;
      }
      if (nread > 0)
***************
*** 633,639 ****
               * EPIPE or ECONNRESET, assume we've lost the backend
               * connection permanently.
               */
!             switch (errno)
              {
  #ifdef EAGAIN
                  case EAGAIN:
--- 633,639 ----
               * EPIPE or ECONNRESET, assume we've lost the backend
               * connection permanently.
               */
!             switch (sockerrno)
              {
  #ifdef EAGAIN
                  case EAGAIN:
***************
*** 668,674 ****
                  default:
                      printfPQExpBuffer(&conn->errorMessage,
                        "pqFlush() --  couldn't send data: errno=%d\n%s\n",
!                                       errno, strerror(errno));
                      /* We don't assume it's a fatal error... */
                      return EOF;
              }
--- 668,674 ----
                  default:
                      printfPQExpBuffer(&conn->errorMessage,
                        "pqFlush() --  couldn't send data: errno=%d\n%s\n",
!                                       sockerrno, strerror(sockerrno));
                      /* We don't assume it's a fatal error... */
                      return EOF;
              }
Index: src/interfaces/libpq/libpq-int.h
===================================================================
RCS file: /home/projects/pgsql/cvsroot/pgsql/src/interfaces/libpq/libpq-int.h,v
retrieving revision 1.33
diff -c -r1.33 libpq-int.h
*** src/interfaces/libpq/libpq-int.h    2001/03/22 04:01:27    1.33
--- src/interfaces/libpq/libpq-int.h    2001/06/04 19:18:23
***************
*** 34,39 ****
--- 34,45 ----
  #include <openssl/err.h>
  #endif

+ #ifndef WIN32
+ #define sockerrno errno
+ #else
+ #define sockerrno (WSAGetLastError() - 10000)
+ #endif
+
  /* libpq supports this version of the frontend/backend protocol.
   *
   * NB: we used to use PG_PROTOCOL_LATEST from the backend pqcomm.h file,

RE: libpq sockets on win32

From
"Jeff Johnson"
Date:
I found this in WinSock.h ... it might shed some light on the problem.


#define WSABASEERR              10000
[snip]

/*
 * Windows Sockets definitions of regular Berkeley error constants
 */
#define WSAEWOULDBLOCK          (WSABASEERR+35)
#define WSAEINPROGRESS          (WSABASEERR+36)



> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> Sent: Monday, June 04, 2001 4:02 PM
> To: jeff@jeffjohnson.net
> Cc: pgsql-interfaces@postgresql.org; pgsql-docs@postgresql.org
> Subject: Re: [INTERFACES] libpq sockets on win32
>
>
> > > I have done exactly that.  I assume fcntl(), ioctl(),
> > > select() use errno
> > > even if used on a socket, while getsockopt(), setsockopt(),
> > socket(),
> > > connect(), getsockname(), send(), recv() use WSAGetLastError.
Is
> > this
> > > list correct?
> >
> > I don't know enough about such things.
>
> The web page wasn't clear about that.
>
> > >
> > > The patch is attached.  Please let me know so I can
> finalize it and
> > > apply it.
> >
> > Couple of changes required to compile on Win32:
> >
> > Change:
> > #define sockerrno WSAGetLastError
> > To:
> > #define sockerrno WSAGetLastError()
> >
>
> OK.
>
> >
> > This has to go back into fe-connect.c, EINPROGRESS isn't defined
in
> > Win32 for some reason..
> > #ifndef WIN32
> >         if (errno == EINPROGRESS || errno == 0)
> > #else
> >         if (WSAGetLastError() == WSAEINPROGRESS)
> > #endif
> >
>
> OK.  Not sure why it wasn't defined, but, oh well.
>
> >
> >
> > I tested it out but I'm getting this error when sending a large
SQL
> > statement (>16k).
> >     cu.execute(sql)
> >   File "c:\python20\pgdb.py", line 189, in execute
> >     self.executemany(operation, (params,))
> >   File "c:\python20\pgdb.py", line 204, in executemany
> >     rows = self.__source.execute(sql)
> > ValueError: pqReadData() --  read() failed: errno=10035
> > Unknown error
>
> That errno is very high, 10035.  If I take a look at my sys/errno.h
on
> BSD/OS, I see:
>
>     #define EAGAIN      35      /* Resource temporarily
> unavailable */
>
> so my guess is that WSAGetLastError() returns errno plus 10,000.
What
> value does your include file set for EAGAIN, 10035 or 35.  If
> it is 35,
> the following patch may work.  It has all your suggested fixes.
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill,
> Pennsylvania 19026
>


Re: libpq sockets on win32

From
Bruce Momjian
Date:
> I found this in WinSock.h ... it might shed some light on the problem.
>
>
> #define WSABASEERR              10000
> [snip]
>
> /*
>  * Windows Sockets definitions of regular Berkeley error constants
>  */
> #define WSAEWOULDBLOCK          (WSABASEERR+35)
> #define WSAEINPROGRESS          (WSABASEERR+36)
>

I modified the patch to use WSABASEERR rather than 10000.

Do your includes define EAGAIN and stuff like that?  Whare are those
values?  If they are the same as WSAEAGAIN, then my patch should work.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: [DOCS] libpq sockets on win32

From
Bruce Momjian
Date:
I have added this thread to TODO.detail/libpq.

> I found that sending more than 16k of data in a SQL statement caused a
> problem.  I was using PyGreSQL but narrowed the error down to
> "\postgresql-7.1.2\src\interfaces\libpq\fe-misc.c" pqReadData() and
> pqFlush().  After some poking around, it looked like recv was setting
> errno to ENOENT for some reason.  I figured out why today.  Win32
> sockets don't set errno at all.  ENOENT was just left in errno from
> some earlier call.
>
> This article describes the problem and work around.
> http://msdn.microsoft.com/library/psdk/winsock/ovrvw3_26ia.htm
>
> I haven't done much C coding in a few years and don't want to break
> other code by blindly doing:
>
> #define errno WSAGetLastError
>
> Does anyone that knows this code want to take a crack at it?
>
> Thanks,
> Jeff
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026