Thread: pg_receivewal.exe unhandled exception in zlib1.dll

pg_receivewal.exe unhandled exception in zlib1.dll

From
Juan José Santamaría Flecha
Date:
Hello,

If you execute pg_receivewal.exe with the option "--compression-method gzip" it will fail with no error. You can see an error in the db log:

2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG:  could not receive data from client: An existing connection was forcibly closed by the remote host.

Tracing the execution you can see:

Unhandled exception at 0x00007FFEA78C1208 (ucrtbase.dll) in An invalid parameter was passed to a function that considers invalid parameters fatal.
  ucrtbase.dll!00007ff8e8ae36a2() Unknown
> zlib1.dll!gz_comp(gz_state * state, int flush) Line 111 C
  zlib1.dll!gz_write(gz_state * state, const void * buf, unsigned __int64 len) Line 235 C
  pg_receivewal.exe!dir_write(void * f, const void * buf, unsigned __int64 count) Line 300 C
  pg_receivewal.exe!ProcessXLogDataMsg(pg_conn * conn, StreamCtl * stream, char * copybuf, int len, unsigned __int64 * blockpos) Line 1150 C
  pg_receivewal.exe!HandleCopyStream(pg_conn * conn, StreamCtl * stream, unsigned __int64 * stoppos) Line 850 C
  pg_receivewal.exe!ReceiveXlogStream(pg_conn * conn, StreamCtl * stream) Line 605 C
  pg_receivewal.exe!StreamLog() Line 636 C
  pg_receivewal.exe!main(int argc, char * * argv) Line 1005 C

The problem comes from the file descriptor passed to gzdopen() in 'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the issue without ifdefing for WIN32. Please find attached a patch for so.

Regards,

Juan José Santamaría Flecha
Attachment

Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Andres Freund
Date:
Hi,

On 2022-02-11 16:33:11 +0100, Juan José Santamaría Flecha wrote:
> If you execute pg_receivewal.exe with the option "--compression-method
> gzip" it will fail with no error. You can see an error in the db log:
> 
> 2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG:
> could not receive data from client: An existing connection was forcibly
> closed by the remote host.
> 
> Tracing the execution you can see:
> 
> Unhandled exception at 0x00007FFEA78C1208 (ucrtbase.dll) in An invalid
> parameter was passed to a function that considers invalid parameters fatal.
>   ucrtbase.dll!00007ff8e8ae36a2() Unknown
> > zlib1.dll!gz_comp(gz_state * state, int flush) Line 111 C
>   zlib1.dll!gz_write(gz_state * state, const void * buf, unsigned __int64
> len) Line 235 C
>   pg_receivewal.exe!dir_write(void * f, const void * buf, unsigned __int64
> count) Line 300 C
>   pg_receivewal.exe!ProcessXLogDataMsg(pg_conn * conn, StreamCtl * stream,
> char * copybuf, int len, unsigned __int64 * blockpos) Line 1150 C
>   pg_receivewal.exe!HandleCopyStream(pg_conn * conn, StreamCtl * stream,
> unsigned __int64 * stoppos) Line 850 C
>   pg_receivewal.exe!ReceiveXlogStream(pg_conn * conn, StreamCtl * stream)
> Line 605 C
>   pg_receivewal.exe!StreamLog() Line 636 C
>   pg_receivewal.exe!main(int argc, char * * argv) Line 1005 C
> 
> The problem comes from the file descriptor passed to gzdopen() in
> 'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the
> issue without ifdefing for WIN32. Please find attached a patch for so.

I hit this as well. The problem is really caused by using a debug build of
postgres vs a production build of zlib. The use different C runtimes, with
different file descriptors. A lot of resources in the windows world are
unfortunately tied to the C runtime and that there can multiple C runtimes in
a single process.

Greetings,

Andres Freund



Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Michael Paquier
Date:
On Fri, Feb 11, 2022 at 07:50:44AM -0800, Andres Freund wrote:
> On 2022-02-11 16:33:11 +0100, Juan José Santamaría Flecha wrote:
>> The problem comes from the file descriptor passed to gzdopen() in
>> 'src/bin/pg_basebackup/walmethods.c'. Using gzopen() instead, solves the
>> issue without ifdefing for WIN32. Please find attached a patch for so.

This looks wrong to me as gzclose() would close() the file descriptor
we pass in via gzdopen(), and some code paths of walmethods.c rely on
this assumption so this would leak fds on repeated errors.

> I hit this as well. The problem is really caused by using a debug build of
> postgres vs a production build of zlib. The use different C runtimes, with
> different file descriptors. A lot of resources in the windows world are
> unfortunately tied to the C runtime and that there can multiple C runtimes in
> a single process.

It may be worth warning about that at build time, or just document the
matter perhaps?  I don't recall seeing any MSIs related to zlib that
have compile with the debug options.
--
Michael

Attachment

Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Juan José Santamaría Flecha
Date:

On Mon, Feb 14, 2022 at 7:31 AM Michael Paquier <michael@paquier.xyz> wrote:
On Fri, Feb 11, 2022 at 07:50:44AM -0800, Andres Freund wrote:

> I hit this as well. The problem is really caused by using a debug build of
> postgres vs a production build of zlib. The use different C runtimes, with
> different file descriptors. A lot of resources in the windows world are
> unfortunately tied to the C runtime and that there can multiple C runtimes in
> a single process.

It may be worth warning about that at build time, or just document the
matter perhaps?  I don't recall seeing any MSIs related to zlib that
have compile with the debug options. 

Building with a mix of debug and release components is not a good practice for issues like this, but is not something that MSVC forbids. If this is something we want to discard altogether, we can check at build time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll.

Regards,

Juan José Santamaría Flecha

Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Andres Freund
Date:
Hi,

On 2022-02-15 11:21:42 +0100, Juan José Santamaría Flecha wrote:
> Building with a mix of debug and release components is not a good practice
> for issues like this, but is not something that MSVC forbids.

It's not a problem if you never need share memory, file descriptors, locales,
... state across DLL boundaries...


> If this is something we want to discard altogether, we can check at build
> time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll.

I think these days it's mostly a question of ucrtbase.dll vs ucrtbased.dll,
right?

FWIW, I've looked at using either vcpkg or conan to more easily install /
build dependencies on windows, in the right debug / release mode.

The former was easier, but unfortunately the zlib, lz4, ... packages don't
include the gzip, lz4 etc binaries right now. That might be easy enough to fix
with an option. It does require building the dependencies locally.

Conan seemed to be a nicer architecture, but there's no builtin way to update
to newer dependency versions, and non-versioned dependencies don't actually
work. It has pre-built dependencies for the common cases.

Greetings,

Andres Freund



Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Juan José Santamaría Flecha
Date:

On Tue, Feb 15, 2022 at 5:25 PM Andres Freund <andres@anarazel.de> wrote:

> If this is something we want to discard altogether, we can check at build
> time with 'dumpbin' to ensure that all dlls share the same vcruntime.dll.

I think these days it's mostly a question of ucrtbase.dll vs ucrtbased.dll,
right?

Either one would do the trick, just to tell apart 'debug' from 'release'.
 
FWIW, I've looked at using either  vcpkg   or conan to more easily install /
build dependencies on windows, in the right debug / release mode.

The former was easier, but unfortunately the zlib, lz4, ... packages don't
include the gzip, lz4 etc binaries right now. That might be easy enough to fix
with an option. It does require building the dependencies locally.

Conan seemed to be a nicer architecture, but there's no builtin way to update
to newer dependency versions, and non-versioned dependencies don't actually
work. It has pre-built dependencies for the common cases.

I find that the repositories that are being currently proposed as a source for the Windows depencies [1], will lead to outdated or stale packages in many cases.

I have been testing vcpkg, and my only grievances have been that some packages were terribly slow to install and a couple were missing: Kerberos (but its own project seems to be active [2]) and ossp-uuid (which is to be expected).

I will have to do some testing with Conan and see how both compare.


Regards,

Juan José Santamaría Flecha
 

Re: pg_receivewal.exe unhandled exception in zlib1.dll

From
Juan José Santamaría Flecha
Date:
Please excuse my late reply.

On Wed, Feb 16, 2022 at 12:20 PM Juan José Santamaría Flecha <juanjo.santamaria@gmail.com> wrote:
On Tue, Feb 15, 2022 at 5:25 PM Andres Freund <andres@anarazel.de> wrote:

FWIW, I've looked at using either  vcpkg   or conan to more easily install /
build dependencies on windows, in the right debug / release mode.

The former was easier, but unfortunately the zlib, lz4, ... packages don't
include the gzip, lz4 etc binaries right now. That might be easy enough to fix
with an option. It does require building the dependencies locally.

Conan seemed to be a nicer architecture, but there's no builtin way to update
to newer dependency versions, and non-versioned dependencies don't actually
work. It has pre-built dependencies for the common cases.

I will have to do some testing with Conan and see how both compare.

I have found a couple of additional issues when trying to build using Conan, all libraries are static and there aren't any prebuilt packages for MSVC 2022 yet. Also for some specific packages:

    - zstd doesn't include the binaries either.
    - gettext packages only provides binaries, no libraries, nor headers.
    - openssl builds the binaries, but is missing some objects in the libraries (error LNK2019: unresolved external symbol).
    - libxml builds the binaries, but is missing some objects in the libraries (error LNK2019: unresolved external symbol).

,Regards,

Juan José Santamaría Flecha