Thread: V14 and later build the backend with -lpthread
I realized $SUBJECT while wondering why my new buildfarm animal chickadee
(NetBSD on gaur's old hardware) fails the plpython tests on v13 and
earlier. After a bit of investigation I realized it *should* be failing,
because neither NetBSD nor Python have done anything about the problem
documented in [1]. The reason it fails to fail in current branches is
that we're now pulling -lpthread into the backend, which AFAICT is an
unintentional side-effect of sloppy autoconfmanship in commits
de91c3b97 / 44bf3d508. We wanted pthread_barrier_wait() for pgbench,
not the backend, but as-committed we'll add -lpthread to LIBS if it
provides pthread_barrier_wait.
Now maybe someday we'll be brave enough to make the backend multithreaded,
but today is not that day, and in the meantime this seems like a rather
dangerous situation. There has certainly been exactly zero analysis
of whether it's safe.
... On the third hand, poking at backends with ldd shows that at
least on Linux, we've been linking the backend with -lpthread for
quite some time, back to 9.4 or so. The new-in-v14 behavior is that
it's getting in there on BSD-ish platforms as well.
Should we try to pull that back out, or just cross our fingers and
hope there's no real problem?
regards, tom lane
[1] https://www.postgresql.org/message-id/flat/25662.1560896200%40sss.pgh.pa.us
On Thu, Aug 25, 2022 at 1:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > I realized $SUBJECT while wondering why my new buildfarm animal chickadee > (NetBSD on gaur's old hardware) fails the plpython tests on v13 and > earlier. After a bit of investigation I realized it *should* be failing, > because neither NetBSD nor Python have done anything about the problem > documented in [1]. The reason it fails to fail in current branches is > that we're now pulling -lpthread into the backend, which AFAICT is an > unintentional side-effect of sloppy autoconfmanship in commits > de91c3b97 / 44bf3d508. We wanted pthread_barrier_wait() for pgbench, > not the backend, but as-committed we'll add -lpthread to LIBS if it > provides pthread_barrier_wait. > > Now maybe someday we'll be brave enough to make the backend multithreaded, > but today is not that day, and in the meantime this seems like a rather > dangerous situation. There has certainly been exactly zero analysis > of whether it's safe. > > ... On the third hand, poking at backends with ldd shows that at > least on Linux, we've been linking the backend with -lpthread for > quite some time, back to 9.4 or so. The new-in-v14 behavior is that > it's getting in there on BSD-ish platforms as well. > > Should we try to pull that back out, or just cross our fingers and > hope there's no real problem? Absent some evidence of a real problem, I vote for crossing our fingers. It would certainly be a very bad idea to start using pthreads willy-nilly in the back end, but the mere presence of the library doesn't seem like a particularly severe issue. I might feel differently if no such version had been released yet, but it's hard to feel like the sky is falling if it's been like this on Linux since 9.4. -- Robert Haas EDB: http://www.enterprisedb.com
On Fri, Aug 26, 2022 at 8:40 AM Robert Haas <robertmhaas@gmail.com> wrote: > On Thu, Aug 25, 2022 at 1:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > I realized $SUBJECT while wondering why my new buildfarm animal chickadee > > (NetBSD on gaur's old hardware) fails the plpython tests on v13 and > > earlier. After a bit of investigation I realized it *should* be failing, > > because neither NetBSD nor Python have done anything about the problem > > documented in [1]. The reason it fails to fail in current branches is > > that we're now pulling -lpthread into the backend, which AFAICT is an > > unintentional side-effect of sloppy autoconfmanship in commits > > de91c3b97 / 44bf3d508. We wanted pthread_barrier_wait() for pgbench, > > not the backend, but as-committed we'll add -lpthread to LIBS if it > > provides pthread_barrier_wait. > > > > Now maybe someday we'll be brave enough to make the backend multithreaded, > > but today is not that day, and in the meantime this seems like a rather > > dangerous situation. There has certainly been exactly zero analysis > > of whether it's safe. > > > > ... On the third hand, poking at backends with ldd shows that at > > least on Linux, we've been linking the backend with -lpthread for > > quite some time, back to 9.4 or so. The new-in-v14 behavior is that > > it's getting in there on BSD-ish platforms as well. > > > > Should we try to pull that back out, or just cross our fingers and > > hope there's no real problem? > > Absent some evidence of a real problem, I vote for crossing our > fingers. It would certainly be a very bad idea to start using pthreads > willy-nilly in the back end, but the mere presence of the library > doesn't seem like a particularly severe issue. I might feel > differently if no such version had been released yet, but it's hard to > feel like the sky is falling if it's been like this on Linux since > 9.4. I suspect we will end up linked against the threading library anyway in real-world builds via --with-XXX (I see that --with-icu has that effect on my FreeBSD system, but I know that details about threading are quite different in NetBSD). I may lack imagination but I'm struggling to see how it could break anything. How should I have done that, by the way? Is the attached the right trick?
Attachment
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Aug 25, 2022 at 1:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> ... On the third hand, poking at backends with ldd shows that at
>> least on Linux, we've been linking the backend with -lpthread for
>> quite some time, back to 9.4 or so. The new-in-v14 behavior is that
>> it's getting in there on BSD-ish platforms as well.
[ further study shows that it's been pulled in on Linux to get sem_init() ]
>> Should we try to pull that back out, or just cross our fingers and
>> hope there's no real problem?
> Absent some evidence of a real problem, I vote for crossing our
> fingers. It would certainly be a very bad idea to start using pthreads
> willy-nilly in the back end, but the mere presence of the library
> doesn't seem like a particularly severe issue. I might feel
> differently if no such version had been released yet, but it's hard to
> feel like the sky is falling if it's been like this on Linux since
> 9.4.
Well, -lpthread on other platforms might have more or different
side-effects than it does on Linux, so I'm not particularly comforted
by that argument. I concede though that the lack of complaints about
v14 is comforting. I'm prepared to do nothing for now; I just wanted
to raise visibility of this point so that if we do come across any
weird pre-vs-post-v14 issues, we think of this as a possible cause.
regards, tom lane
Thomas Munro <thomas.munro@gmail.com> writes:
> I suspect we will end up linked against the threading library anyway
> in real-world builds via --with-XXX (I see that --with-icu has that
> effect on my FreeBSD system, but I know that details about threading
> are quite different in NetBSD). I may lack imagination but I'm
> struggling to see how it could break anything.
Yeah, there are plenty of situations where you end up with thread
support present somehow. So it may be a lost cause. I was mostly
concerned about this because it seemed like an unintentional change.
(I'm also still struggling to explain why mamba, with the *exact*
same NetBSD code on a different hardware platform, isn't showing
the same failures as chickadee. More news if I figure that out.)
> How should I have done that, by the way? Is the attached the right trick?
I think that'd do for preventing side-effects on LIBS, but I'm not
sure if we'd have to back-fill something in pgbench's link options.
Anyway, as I said to Robert, I'm content to watch and wait for now.
regards, tom lane
I wrote:
> (I'm also still struggling to explain why mamba, with the *exact*
> same NetBSD code on a different hardware platform, isn't showing
> the same failures as chickadee. More news if I figure that out.)
Hah: I left --with-libxml out of chickadee's configuration, because
libxml2 seemed to have some problems on that platform, and that is
what is pulling in libpthread on mamba:
$ ldd /usr/pkg/lib/libxml2.so
/usr/pkg/lib/libxml2.so:
-lz.1 => /usr/lib/libz.so.1
-lc.12 => /usr/lib/libc.so.12
-llzma.2 => /usr/lib/liblzma.so.2
-lpthread.1 => /lib/libpthread.so.1
-lm.0 => /usr/lib/libm.so.0
-lgcc_s.1 => /lib/libgcc_s.so.1
Reinforces your point about real-world builds, I suppose.
For the moment I'll just disable testing plpython pre-v14 on
chickadee.
regards, tom lane
Hi, On 2022-08-25 17:04:37 -0400, Tom Lane wrote: > (I'm also still struggling to explain why mamba, with the *exact* > same NetBSD code on a different hardware platform, isn't showing > the same failures as chickadee. More news if I figure that out.) I'd guess it's because of the different dependencies that are enabled. On my netbsd VM libxml2 pulls in -lpthread, for example. We add xml2's dependencies to LIBS, so if that's enabled, we end up indirectly pulling in libxml2 in as well. > > How should I have done that, by the way? Is the attached the right trick? > > I think that'd do for preventing side-effects on LIBS, but I'm not > sure if we'd have to back-fill something in pgbench's link options. > Anyway, as I said to Robert, I'm content to watch and wait for now. Given that linking in pthreads support fixes things, that seems the right course... I wonder if we shouldn't even be more explicit about it and just add it - who knows what extension libraries pull in. It'd not be good if we end up with non-reentrant versions of functions just because initially the backend isn't threaded etc. Greetings, Andres Freund