Re: Unknown winsock error 10061 - Mailing list pgsql-bugs

Dave Page <dpage@pgadmin.org> wrote:
 > 2009/7/7 Wojciech Strzałka <wstrzalka@gmail.com>:
 > >
 > >  Here ( http://www.codelabs.pl/_varia/pg.zip ) is detailed info from
 > >  my machine (PostgreSQL 8.3.6, compiled by Visual C++ build 1400):
 > >
 > >  - detailed log file with several memory attach problems
 > >  - process activity log file (created by Process Monitor from SysInternals)
 > >  - dll's loaded by postgres.exe (snapshot only)
 > >
 > >  Maybe by comparing log entries you will be able to tell smth.
 >
 > I can't spot anything obviously wrong (other than the original error
 > of course). My only suggestion right now is that we try putting a
 > retry loop in PGSharedMemoryReAttach() and see if the error is a
 > transient thing.
 >
 > Any other ideas?

Hello,

When failed to reattach, how about the memory regions ?

For instance, do Sleep() when reattach failed, and check the memory by VMMap.
(from SysInternals)
# Of course, it becomes only for debugging.

I think the region to be used as Shared Memory might be used as Heap.
compare with the parent process.

I don't know why the Heap region moves.
 - the order of which DLL is loaded ?
 - Heap size is changed ?

I think this error often occurs when postmater receives a lot of requests.
In our environment, we can reproduce error by following tests.
# AMD Opteron 850 (4 processor) + Windows Server 2008

(Windows server)
postgresql.conf
  max_connections = 300

(*nix client)
export PGHOST=WINDOWS
export PGUSER=postgres
while :; do
  pgbench -n -c 250 || break
done
# maybe fail in 3 to 5 loops


Though the topic changes a little, we are testing about this problem.

In the past, there was the thread about this problem,
and it suggested using VirtualAllocEx().
(But no patch was posted, right?)
http://archives.postgresql.org/pgsql-general/2007-08/msg01592.php

So I tries using VirtualAllocEx().
By this fix, the problem doesn't occur in testing by pgbench.

However, I cannot explain why the memory region was changed,
and cannot guarantee that all problems are solved.

FYI, my patch is put up.
Sorry for large patch, it has too many debug codes.

* postmaster.c
  In internal_forkexec(), reserve shared memory region before ResumeThread().

* win32_shmem.c
  new function to do VirtualAllocEx()
  PGSharedMemoryReAttach() free reserved region before MapViewOfFileEx()

* DEBUG code
  if define DEBUG_VQ, outputs info about memory regions to stderr.
  (Sorry, it is not so useful...)

  if define DEBUG_VQ_DONT_RESERV,
   don't call VirtualAllocEx()/VirtualFree(), so it works as original
   with debug-outs.
   This do Sleep() when reattach fail.


regards,

--
Tsutomu Yamada
SRA OSS, Inc. Japan

Index: src/backend/port/win32_shmem.c
===================================================================
RCS file: /mnt/prj/pg/cvsmirror/pg/pgsql/src/backend/port/win32_shmem.c,v
retrieving revision 1.11
diff -c -r1.11 win32_shmem.c
*** src/backend/port/win32_shmem.c    11 Jun 2009 14:49:00 -0000    1.11
--- src/backend/port/win32_shmem.c    8 Jul 2009 11:52:14 -0000
***************
*** 18,23 ****
--- 18,30 ----

  unsigned long UsedShmemSegID = 0;
  void       *UsedShmemSegAddr = NULL;
+ static Size UsedShmemSegSize = 0;
+
+ //#define DEBUG_VQ
+ #ifdef DEBUG_VQ
+ //#define DEBUG_VQ_DONT_RESERV
+ void DumpVirtualQuery(HANDLE);
+ #endif

  static void pgwin32_SharedMemoryDelete(int status, Datum shmId);

***************
*** 183,188 ****
--- 190,200 ----
                  (errmsg("pre-existing shared memory block is still in use"),
                   errhint("Check if there are any old server processes still running, and terminate them.")));

+ #ifdef DEBUG_VQ
+     fprintf(stderr, "before MapViewOfFileEx: hmap=%p (size=%lu, name=%s)\n",
+             hmap, (unsigned long) size, szShareMem);
+     DumpVirtualQuery(NULL);
+ #endif
      free(szShareMem);

      /*
***************
*** 233,240 ****
--- 245,258 ----

      /* Save info for possible future use */
      UsedShmemSegAddr = memAddress;
+     UsedShmemSegSize = size;
      UsedShmemSegID = (unsigned long) hmap2;

+ #ifdef DEBUG_VQ
+     fprintf(stderr, "PGSharedMemoryCreate ok: key %ld, SegAddr=%p\n",
+             UsedShmemSegID, UsedShmemSegAddr);
+     DumpVirtualQuery(NULL);
+ #endif
      return hdr;
  }

***************
*** 257,266 ****
--- 275,361 ----
      Assert(UsedShmemSegAddr != NULL);
      Assert(IsUnderPostmaster);

+ #ifndef DEBUG_VQ_DONT_RESERV
+     {
+         /* release region of memory
+          * reserved by parant process
+          */
+ #ifdef DEBUG_VQ_XXX /* too verbose */
+         fprintf(stderr, "--reattach: before vfree\n");
+         DumpVirtualQuery(NULL);
+ #endif
+         if (VirtualFree(UsedShmemSegAddr, UsedShmemSegSize, MEM_RELEASE) == 0)
+         {
+ #ifndef DEBUG_VQ
+             elog(WARNING, "failed to release pre-reserved memory: %lu",
+                  GetLastError());
+ #else /* verbose errer message */
+             LPVOID lpMsgBuf;
+             unsigned long ecode;
+             FormatMessage(
+                 FORMAT_MESSAGE_ALLOCATE_BUFFER |
+                 FORMAT_MESSAGE_FROM_SYSTEM |
+                 FORMAT_MESSAGE_IGNORE_INSERTS,
+                 NULL,
+                 ecode = GetLastError(),
+                 MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), //
+                 (LPTSTR) &lpMsgBuf,
+                 0,
+                 NULL
+                 );
+             fprintf(stderr, "== VirtualFree fail: %lu: %s\n", ecode, lpMsgBuf);
+             LocalFree(lpMsgBuf);
+ #endif
+         }
+ #ifdef DEBUG_VQ_XXX /* too verbose */
+         fprintf(stderr, "--reattach: after vfree\n");
+         DumpVirtualQuery(NULL);
+ #endif
+     }
+ #endif
+
      hdr = (PGShmemHeader *) MapViewOfFileEx((HANDLE) UsedShmemSegID, FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 0,
UsedShmemSegAddr);
      if (!hdr)
+ #ifndef DEBUG_VQ
          elog(FATAL, "could not reattach to shared memory (key=%d, addr=%p): %lu",
               (int) UsedShmemSegID, UsedShmemSegAddr, GetLastError());
+ #else
+     {
+         LPVOID lpMsgBuf;
+         unsigned long ecode;
+         FormatMessage(
+             FORMAT_MESSAGE_ALLOCATE_BUFFER |
+             FORMAT_MESSAGE_FROM_SYSTEM |
+             FORMAT_MESSAGE_IGNORE_INSERTS,
+             NULL,
+             ecode = GetLastError(),
+             MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), //
+             (LPTSTR) &lpMsgBuf,
+             0,
+             NULL
+             );
+         fprintf(stderr, "ReAttach fail: PID=%d, hdr(new)%p, Used %p\n",
+                 getpid(), hdr, UsedShmemSegAddr);
+         DumpVirtualQuery(NULL);
+ #ifdef DEBUG_VQ_DONT_RESERV
+         {
+             /* XXX: to use monitor utilities */
+             int ii;
+             for (ii = 0; ii < 10; ii++) {
+                 fprintf(stderr, "--sleeping: reattach fail: PID=%d SegAddr=%p\n", getpid(), UsedShmemSegAddr);
+                 Sleep(1000 * 60);
+             }
+         }
+ #endif
+         elog(FATAL, "could not reattach to shared memory (key=%d, addr=%p): %lu: %s",
+              (int) UsedShmemSegID, UsedShmemSegAddr, ecode, lpMsgBuf);
+         LocalFree(lpMsgBuf);
+     }
+ #endif
+ #ifdef DEBUG_VQ_XXX /* too verbose */
+     fprintf(stderr, "--reattach: after mapview\n");
+     DumpVirtualQuery(NULL);
+ #endif
      if (hdr != origUsedShmemSegAddr)
          elog(FATAL, "reattaching to shared memory returned unexpected address (got %p, expected %p)",
               hdr, origUsedShmemSegAddr);
***************
*** 302,304 ****
--- 397,621 ----
      if (!CloseHandle((HANDLE) DatumGetInt32(shmId)))
          elog(LOG, "could not close handle to shared memory: %lu", GetLastError());
  }
+
+ /*
+  * pgwin32_ReserveSharedMemory(HANDLE pChild)
+  * Reserve shared memory area,
+  * BEFORE child process allocates memory for DLL and/or others.
+  */
+ void
+ pgwin32_ReserveSharedMemory(HANDLE pChild)
+ {
+     void *memAddress;
+
+ #ifdef DEBUG_VQ
+     static int debugcnt = 10;
+     int do_dumpVQ = 0;
+
+ #ifdef DEBUG_VQ_DONT_RESERV
+     return; /* do nothing */
+ #endif
+
+     /* dump Nth child */
+     if (debugcnt > 0) {
+         if (--debugcnt == 0) {
+             do_dumpVQ = 1;
+         }
+     }
+
+     if (do_dumpVQ) {
+         fprintf(stderr, "== before VirtualAllocEx %p: SegAddr=%p size=%lu\n",
+                 pChild, UsedShmemSegAddr, (unsigned long) UsedShmemSegSize);
+         DumpVirtualQuery(pChild);
+     }
+ #endif
+
+     Assert(UsedShmemSegAddr != NULL);
+     memAddress = VirtualAllocEx(pChild, UsedShmemSegAddr, UsedShmemSegSize,
+                                 MEM_RESERVE, PAGE_READWRITE);
+     if (memAddress == NULL) {
+ #ifndef DEBUG_VQ
+         elog(LOG, "could not reserve shared memory area: error code %lu",
+              GetLastError());
+ #else
+         LPVOID lpMsgBuf;
+         unsigned long ecode;
+         FormatMessage(
+             FORMAT_MESSAGE_ALLOCATE_BUFFER |
+             FORMAT_MESSAGE_FROM_SYSTEM |
+             FORMAT_MESSAGE_IGNORE_INSERTS,
+             NULL,
+             ecode = GetLastError(),
+             MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), //
+             (LPTSTR) &lpMsgBuf,
+             0,
+             NULL
+             );
+         fprintf(stderr, "== VirtualAllocEx fail: %lu: %s\n", ecode, lpMsgBuf);
+         LocalFree(lpMsgBuf);
+ #endif
+     }
+
+ #ifdef DEBUG_VQ
+     if (do_dumpVQ) {
+         fprintf(stderr, "== after VirtualAllocEx: mem=%p\n", memAddress);
+         DumpVirtualQuery(pChild);
+     }
+ #endif
+ }
+
+ #ifdef DEBUG_VQ
+ /* debug
+  */
+ struct flg2str {
+     DWORD flg;
+     char *str;
+ };
+
+ /* XXX: all strings has same width */
+ static struct flg2str protection[] = {
+     { PAGE_EXECUTE,                "EXEC " },
+     { PAGE_EXECUTE_READ,        "EX_RO" },
+     { PAGE_EXECUTE_READWRITE,    "EX_RW" },
+     { PAGE_EXECUTE_WRITECOPY,    "EX_WC" },
+     { PAGE_NOACCESS,            "NOACC" },
+     { PAGE_READONLY,            "RONLY" },
+     { PAGE_READWRITE,            "RW   " },
+     { PAGE_WRITECOPY,            "WCOPY" },
+     { 0, NULL }
+ };
+
+ static struct flg2str p_modifiers[] = {
+     { PAGE_GUARD, "-PG" },
+     { PAGE_NOCACHE, "-NC" },
+     { PAGE_WRITECOMBINE, "-WC" },
+     { 0, NULL }
+ };
+
+ static struct flg2str page_state[] = {
+     { MEM_COMMIT,    "commit " },
+     { MEM_FREE,        "free   " },
+     { MEM_RESERVE,    "reserve" },
+     { 0, NULL }
+ };
+
+ static struct flg2str page_type[] = {
+     { MEM_IMAGE,    "image  " },
+     { MEM_MAPPED,    "mapped " },
+     { MEM_PRIVATE,    "private" },
+     { 0, NULL }
+ };
+
+ static void print_flg(struct flg2str f2s[], DWORD flgval)
+ {
+     int i;
+
+     for (i = 0; f2s[i].str != NULL; i++) {
+         if (f2s[i].flg == flgval) {
+             fprintf(stderr, " %s", f2s[i].str);
+             break;
+         }
+     }
+     if (f2s[i].str == NULL) {
+         fprintf(stderr, " ?%x", flgval);
+     }
+     return;
+ }
+
+ static void DumpVirtualQuery(HANDLE hProcess)
+ {
+     SIZE_T length;
+     LPCBYTE addr;
+     LPCBYTE prevBase = NULL;
+     MEMORY_BASIC_INFORMATION infobuff;
+     SIZE_T dwLength = sizeof(infobuff);
+     int i = 0;
+
+     fprintf(stderr, "-VQ- %p\n", hProcess);
+     addr = NULL;
+
+     for (;;) {
+         if (hProcess != NULL)
+             length = VirtualQueryEx(hProcess, addr, &infobuff, dwLength);
+         else
+             length = VirtualQuery(addr, &infobuff, dwLength);
+
+         if (length <= 0)
+             break;
+
+         if (length != dwLength) {
+             fprintf(stderr, "?VQ fail?");
+         }
+
+ #ifdef _WIN64
+ #define PADDR(x) fprintf(stderr, "%011llX", (unsigned long long) (x))
+ #define PSAME()  fprintf(stderr, " |         ")
+ #else
+ #define PADDR(x) fprintf(stderr, "%p", (x))
+ #define PSAME()  fprintf(stderr, " |      ")
+ #endif
+
+         /* addr, size */
+         PADDR(infobuff.BaseAddress);
+         fprintf(stderr, "-");
+         PADDR((LPCBYTE) infobuff.BaseAddress + infobuff.RegionSize - 1);
+         fprintf(stderr, " ");
+
+         if (infobuff.AllocationBase == NULL) {
+             /* not mapped */
+             fprintf(stderr, "-nomap-");
+             /* assert */
+             if (infobuff.AllocationProtect != 0 ||
+                 infobuff.State != MEM_FREE ||
+                 infobuff.Protect != PAGE_NOACCESS ||
+                 !(infobuff.Type == 0 || infobuff.Type == MEM_IMAGE)) {
+                 fprintf(stderr, " ??? ap=%x, s=%x, p=%x, t=%x",
+                         infobuff.AllocationProtect,
+                         infobuff.State,
+                         infobuff.Protect,
+                         infobuff.Type);
+             }
+         } else {
+             if ((LPCBYTE) infobuff.AllocationBase == prevBase) {
+                 /* same allocation base as above */
+                 PSAME();
+             } else {
+                 prevBase = (LPCBYTE) infobuff.AllocationBase;
+                 PADDR(infobuff.AllocationBase);
+             }
+
+             /* flags */
+             print_flg(protection, (infobuff.AllocationProtect & 0xff));
+             if (infobuff.AllocationProtect & ~0xff) {
+                 print_flg(p_modifiers, (infobuff.AllocationProtect & ~0xff));
+             }
+
+             print_flg(page_state, infobuff.State);
+
+             print_flg(protection, (infobuff.Protect & 0xff));
+             if (infobuff.Protect & ~0xff) {
+                 print_flg(p_modifiers, (infobuff.Protect & ~0xff));
+             }
+
+             print_flg(page_type, infobuff.Type);
+         }
+         fprintf(stderr, "\n");
+
+ #if 0
+         fprintf(stderr, "%p %p a%3x %p s%5x p%3x t%7x\n",
+                 infobuff.BaseAddress,
+                 infobuff.AllocationBase,
+                 infobuff.AllocationProtect,
+                 infobuff.RegionSize,
+                 infobuff.State,
+                 infobuff.Protect,
+                 infobuff.Type
+             );
+ #endif
+         addr = (LPCBYTE) infobuff.BaseAddress + infobuff.RegionSize;
+         if (++i >= 500) /* failsafe */
+             break;
+     }
+     fprintf(stderr, "-VQ-end\n");
+ }
+ #endif /*DEBUG_VQ*/
Index: src/backend/postmaster/postmaster.c
===================================================================
RCS file: /mnt/prj/pg/cvsmirror/pg/pgsql/src/backend/postmaster/postmaster.c,v
retrieving revision 1.583
diff -c -r1.583 postmaster.c
*** src/backend/postmaster/postmaster.c    26 Jun 2009 20:29:04 -0000    1.583
--- src/backend/postmaster/postmaster.c    7 Jul 2009 09:54:24 -0000
***************
*** 3645,3650 ****
--- 3645,3657 ----
          elog(LOG, "could not close handle to backend parameter file: error code %d",
               (int) GetLastError());

+     {
+         /* reserve shared memory area before ResumeThread() */
+         /* XXX: if it fail ? */
+         extern void pgwin32_ReserveSharedMemory(HANDLE);
+         pgwin32_ReserveSharedMemory(pi.hProcess);
+     }
+
      /*
       * Now that the backend variables are written out, we start the child
       * thread so it can start initializing while we set up the rest of the