Re: multithreaded zstd backup compression for client and server - Mailing list pgsql-hackers

From Justin Pryzby
Subject Re: multithreaded zstd backup compression for client and server
Date
Msg-id 20220323230701.GY28503@telsasoft.com
Whole thread Raw
In response to multithreaded zstd backup compression for client and server  (Robert Haas <robertmhaas@gmail.com>)
Responses Re: multithreaded zstd backup compression for client and server  (Robert Haas <robertmhaas@gmail.com>)
Re: multithreaded zstd backup compression for client and server  (Andres Freund <andres@anarazel.de>)
List pgsql-hackers
On Wed, Mar 23, 2022 at 04:34:04PM -0400, Robert Haas wrote:
> be, spawning threads inside the PostgreSQL backend. Short of cats and
> dogs living together, it's hard to think of anything more terrifying,
> because the PostgreSQL backend is very much not thread-safe. However,
> a lot of the things we usually worry about when people make noises
> about using threads in the backend don't apply here, because the
> threads are hidden away behind libzstd interfaces and can't execute
> any PostgreSQL code. Therefore, I think it might be safe to just ...
> turn this on. One reason I think that is that this whole approach was
> recommended to me by Andres ... but that's not to say that there
> couldn't be problems.  I worry a bit that the mere presence of threads
> could in some way mess things up, but I don't know what the mechanism
> for that would be, and I don't want to postpone shipping useful
> features based on nebulous fears.

Note that the PGDG .RPMs and .DEBs are already linked with pthread, via
libxml => liblzma.

$ ldd /usr/pgsql-14/bin/postgres |grep xm
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007faab984e000)
$ objdump -p /lib64/libxml2.so.2 |grep NEED
  NEEDED               libdl.so.2
  NEEDED               libz.so.1
  NEEDED               liblzma.so.5
  NEEDED               libm.so.6
  NEEDED               libc.so.6
  VERNEED              0x0000000000019218
  VERNEEDNUM           0x0000000000000005
$ objdump -p /lib64/liblzma.so.5 |grep NEED
  NEEDED               libpthread.so.0



Did you try this on windows at all ?  It's probably no surprise that zstd
implements threading differently there.



pgsql-hackers by date:

Previous
From: Jacob Champion
Date:
Subject: Re: [PATCH] Expose port->authn_id to extensions and triggers
Next
From: Andres Freund
Date:
Subject: Re: multithreaded zstd backup compression for client and server