Thread: [PATCH 1/1] Fix compilation on mac with Xcode >= 11.4.
It would appear weak symbol linking is not handled properly without
'isysroot' parameter passed to linker.
Fixes:
Undefined symbols for architecture x86_64:
"___darwin_check_fd_set_overflow", referenced from:
_ClientAuthentication in auth.o
_pgstat_init in pgstat.o
_ServerLoop in postmaster.o
ld: symbol(s) not found for architecture x86_64
See:
https://github.com/qt/qtwebengine-chromium/commit/d5c4b6230b7f915f6e044e230c0c575249938400
---
configure | 4 +++-
configure.ac | 4 +++-
src/template/darwin | 2 ++
3 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/configure b/configure
index ace4ed5dec..92b089907e 100755
--- a/configure
+++ b/configure
@@ -19041,7 +19041,6 @@ if test "$with_readline" = yes; then
else
link_test_func=exit
fi
-
if test "$PORTNAME" = "darwin"; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC supports -Wl,-dead_strip_dylibs" >&5
$as_echo_n "checking whether $CC supports -Wl,-dead_strip_dylibs... " >&6; }
@@ -19194,6 +19193,9 @@ _ACEOF
# we've finished all configure checks that depend on CPPFLAGS.
if test x"$PG_SYSROOT" != x; then
CPPFLAGS=`echo "$CPPFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+ if test "$PORTNAME" = "darwin"; then
+ LDFLAGS=`echo "$LDFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+ fi
fi
diff --git a/configure.ac b/configure.ac
index 5b91c83fd0..a82cbbc147 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2326,7 +2326,6 @@ if test "$with_readline" = yes; then
else
link_test_func=exit
fi
-
if test "$PORTNAME" = "darwin"; then
PGAC_PROG_CC_LDFLAGS_OPT([-Wl,-dead_strip_dylibs], $link_test_func)
elif test "$PORTNAME" = "openbsd"; then
@@ -2362,6 +2361,9 @@ AC_SUBST(PG_VERSION_NUM)
# we've finished all configure checks that depend on CPPFLAGS.
if test x"$PG_SYSROOT" != x; then
CPPFLAGS=`echo "$CPPFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+ if test "$PORTNAME" = "darwin"; then
+ LDFLAGS=`echo "$LDFLAGS" | sed -e "s| $PG_SYSROOT | \\\$(PG_SYSROOT) |"`
+ fi
fi
AC_SUBST(PG_SYSROOT)
diff --git a/src/template/darwin b/src/template/darwin
index f4d4e9d7cf..83d20a8cdd 100644
--- a/src/template/darwin
+++ b/src/template/darwin
@@ -11,6 +11,8 @@ fi
if test x"$PG_SYSROOT" != x"" ; then
if test -d "$PG_SYSROOT" ; then
CPPFLAGS="-isysroot $PG_SYSROOT $CPPFLAGS"
+ # Prevent undefined symbols errors for weak linked symbols.
+ LDFLAGS="-isysroot $PG_SYSROOT $LDFLAGS"
else
PG_SYSROOT=""
fi
--
2.29.2
James Hilliard <james.hilliard1@gmail.com> writes:
> It would appear weak symbol linking is not handled properly without
> 'isysroot' parameter passed to linker.
Nobody else has reported this problem, so maybe you should tell us
how to reproduce it before you suggest a build process change.
(And yes, we have buildfarm members testing Xcode > 11.4 ...)
regards, tom lane
On Thu, Nov 19, 2020 at 5:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > James Hilliard <james.hilliard1@gmail.com> writes: > > It would appear weak symbol linking is not handled properly without > > 'isysroot' parameter passed to linker. > > Nobody else has reported this problem, so maybe you should tell us > how to reproduce it before you suggest a build process change. Hmm, maybe it's a more recent issue then, I took the version number from the qt patch assuming it was the same issue, I hit it trying to build master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7. > > (And yes, we have buildfarm members testing Xcode > 11.4 ...) > > regards, tom lane
I think I figured it out, it only happens on systems where the Xcode version is newer than the OS it seems. See: https://forum.unity.com/threads/il2cpp-macstandalone-and-xcode-11-4.855187/ So looks like the failure happens due to the system sysroot or something like that missing symbols from the Xcode sysroot, so I think this patch is still correct in that case as it forces the use of the Xcode sysroot for linking which should be what we want as we want it to match sysroot passed to the compiler. On Thu, Nov 19, 2020 at 5:45 PM James Hilliard <james.hilliard1@gmail.com> wrote: > > On Thu, Nov 19, 2020 at 5:40 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > > > James Hilliard <james.hilliard1@gmail.com> writes: > > > It would appear weak symbol linking is not handled properly without > > > 'isysroot' parameter passed to linker. > > > > Nobody else has reported this problem, so maybe you should tell us > > how to reproduce it before you suggest a build process change. > Hmm, maybe it's a more recent issue then, I took the version number > from the qt patch assuming it was the same issue, I hit it trying to build > master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7. > > > > (And yes, we have buildfarm members testing Xcode > 11.4 ...) > > > > regards, tom lane
James Hilliard <james.hilliard1@gmail.com> writes:
> Hmm, maybe it's a more recent issue then, I took the version number
> from the qt patch assuming it was the same issue, I hit it trying to build
> master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.
Hm, maybe you're using some unusual configure options or weird build
environment?
The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
Maybe you're found some corner case in between those, but I guess it's
more likely due to a configuration choice.
regards, tom lane
On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> James Hilliard <james.hilliard1@gmail.com> writes:
> > Hmm, maybe it's a more recent issue then, I took the version number
> > from the qt patch assuming it was the same issue, I hit it trying to build
> > master on Xcode 12.2 Build version 12B45b on Catalina version 10.15.7.
>
> Hm, maybe you're using some unusual configure options or weird build
> environment?
>
> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
> Maybe you're found some corner case in between those, but I guess it's
> more likely due to a configuration choice.
I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7.
This looks to be the function using the missing symbols, called by
FD_SET in postgres:
__header_always_inline int
__darwin_check_fd_set(int _a, const void *_b)
{
#ifdef __clang__
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wunguarded-availability-new"
#endif
if ((uintptr_t)&__darwin_check_fd_set_overflow != (uintptr_t) 0) {
#if defined(_DARWIN_UNLIMITED_SELECT) || defined(_DARWIN_C_SOURCE)
return __darwin_check_fd_set_overflow(_a, _b, 1);
#else
return __darwin_check_fd_set_overflow(_a, _b, 0);
#endif
} else {
return 1;
}
#ifdef __clang__
#pragma clang diagnostic pop
#endif
}
Which is located here:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include/sys/_types/_fd_def.h
The header file indicates the missing symbols are only available in 11.0 and up:
int __darwin_check_fd_set_overflow(int, const void *, int)
__API_AVAILABLE(macosx(11.0), ios(14.0), tvos(14.0), watchos(7.0));
So I think it's fairly clear what's happening now, the linker would normally use
the system libraries to determine that those missing symbols are
weakly linked, but since
the base OS is too old that fails unless we tell the linker where to
look(in the SDK).
>
> regards, tom lane
James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7)
>> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel).
>> Maybe you're found some corner case in between those, but I guess it's
>> more likely due to a configuration choice.
> I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7.
To check this, I updated my laptop (still on Catalina 10.15.7) to latest
Xcode, 12.2 (12B45b), from 12.0 --- and behold, I could duplicate it:
Undefined symbols for architecture x86_64:
"___darwin_check_fd_set_overflow", referenced from:
_ClientAuthentication in auth.o
_pgstat_init in pgstat.o
_ServerLoop in postmaster.o
ld: symbol(s) not found for architecture x86_64
However ... it then occurred to me to blow away my ccache and accache,
and after rebuilding from scratch, everything's fine. So, are you
using ccache?
regards, tom lane
On Thu, Nov 19, 2020 at 7:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > James Hilliard <james.hilliard1@gmail.com> writes: > > On Thu, Nov 19, 2020 at 6:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> The cases we've got in the buildfarm are Xcode 12.0 on Catalina (10.15.7) > >> and Xcode 12.2 on Big Sur (11.0.1 ... although that one is ARM not Intel). > >> Maybe you're found some corner case in between those, but I guess it's > >> more likely due to a configuration choice. > > > I guess to verify one could try compiling with Xcode 12.2 on catalina 10.15.7. > > To check this, I updated my laptop (still on Catalina 10.15.7) to latest > Xcode, 12.2 (12B45b), from 12.0 --- and behold, I could duplicate it: > > Undefined symbols for architecture x86_64: > "___darwin_check_fd_set_overflow", referenced from: > _ClientAuthentication in auth.o > _pgstat_init in pgstat.o > _ServerLoop in postmaster.o > ld: symbol(s) not found for architecture x86_64 > > However ... it then occurred to me to blow away my ccache and accache, > and after rebuilding from scratch, everything's fine. So, are you > using ccache? Don't think so, I first hit this issue on a clean clone of master and I don't think I even have ccache or accache(not familiar with this variant) installed at all. Are you sure something else didn't get changed when you cleared the caches? Is configure using the right SDK directory? > > regards, tom lane
I wrote: > However ... it then occurred to me to blow away my ccache and accache, > and after rebuilding from scratch, everything's fine. So, are you > using ccache? Oh, scratch that, I fat-fingered the experiment somehow. The issue is still there. Still, I'm hesitant to apply the fix you suggest, because of the law of unintended consequences. In particular, I'm afraid that using -isysroot in the link step might result in executables that are bound to that sysroot and will not work if it's not there --- thus causing a problem for packagers trying to distribute PG to non-developers. We had a great deal of trouble in the initial experiments with -isysroot, cf https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us so I'm not that eager to mess with it. I'm inclined to think that this is more a case of "Apple broke ABI and they ought to fix that". regards, tom lane
On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > I wrote: > > However ... it then occurred to me to blow away my ccache and accache, > > and after rebuilding from scratch, everything's fine. So, are you > > using ccache? > > Oh, scratch that, I fat-fingered the experiment somehow. The issue > is still there. Still, I'm hesitant to apply the fix you suggest, > because of the law of unintended consequences. In particular, I'm > afraid that using -isysroot in the link step might result in executables > that are bound to that sysroot and will not work if it's not there > --- thus causing a problem for packagers trying to distribute PG > to non-developers. Since apple doesn't actually put real dylib's in the SDK path I think we're fine: $ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -name "*.dylib" $ Instead they use .tbd or text-based stub files there which appear to provide enough info for the linker to determine that the undefined symbols are weak linked. And since the linker doesn't actually link against the .tbd file paths in the way it would a dylib I don't think the SDK path could get used accidentally. > > We had a great deal of trouble in the initial experiments with > -isysroot, cf > > https://www.postgresql.org/message-id/flat/20840.1537850987%40sss.pgh.pa.us > > so I'm not that eager to mess with it. I'm inclined to think that > this is more a case of "Apple broke ABI and they ought to fix that". This seems to be a longstanding issue, I expect they don't care much about autotools compared with their normal Xcode project build system. So probably better to just give this a try and see how it goes as it's likely to keep popping up otherwise. > > regards, tom lane
James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Oh, scratch that, I fat-fingered the experiment somehow. The issue
>> is still there. Still, I'm hesitant to apply the fix you suggest,
>> because of the law of unintended consequences. In particular, I'm
>> afraid that using -isysroot in the link step might result in executables
>> that are bound to that sysroot and will not work if it's not there
>> --- thus causing a problem for packagers trying to distribute PG
>> to non-developers.
> Since apple doesn't actually put real dylib's in the SDK path I think
> we're fine:
> $ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
> -name "*.dylib"
True. Also, while actual documentation on -isysroot seems to be damn
near nonexistent, all the example usages I can find on the net appear
to put it into both compile and link steps. So maybe we're just doing
it wrong and happened to get away with that this long.
> This seems to be a longstanding issue, I expect they don't care much about
> autotools compared with their normal Xcode project build system. So probably
> better to just give this a try and see how it goes as it's likely to
> keep popping up otherwise.
Yeah, the one saving grace here is that, unlike the last time, we're not
hard up against a release deadline. If we push this now we can hope to
get a fair amount of testing from PG developers before we have to ship
it.
regards, tom lane
On Thu, Nov 19, 2020 at 9:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > James Hilliard <james.hilliard1@gmail.com> writes: > > On Thu, Nov 19, 2020 at 7:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> Oh, scratch that, I fat-fingered the experiment somehow. The issue > >> is still there. Still, I'm hesitant to apply the fix you suggest, > >> because of the law of unintended consequences. In particular, I'm > >> afraid that using -isysroot in the link step might result in executables > >> that are bound to that sysroot and will not work if it's not there > >> --- thus causing a problem for packagers trying to distribute PG > >> to non-developers. > > > Since apple doesn't actually put real dylib's in the SDK path I think > > we're fine: > > $ find /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk > > -name "*.dylib" > > True. Also, while actual documentation on -isysroot seems to be damn > near nonexistent, all the example usages I can find on the net appear > to put it into both compile and link steps. So maybe we're just doing > it wrong and happened to get away with that this long. Yeah, pretty sure that's what happened from what I'm seeing. > > > This seems to be a longstanding issue, I expect they don't care much about > > autotools compared with their normal Xcode project build system. So probably > > better to just give this a try and see how it goes as it's likely to > > keep popping up otherwise. > > Yeah, the one saving grace here is that, unlike the last time, we're not > hard up against a release deadline. If we push this now we can hope to > get a fair amount of testing from PG developers before we have to ship > it. > > regards, tom lane
James Hilliard <james.hilliard1@gmail.com> writes:
> On Thu, Nov 19, 2020 at 9:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> True. Also, while actual documentation on -isysroot seems to be damn
>> near nonexistent, all the example usages I can find on the net appear
>> to put it into both compile and link steps. So maybe we're just doing
>> it wrong and happened to get away with that this long.
> Yeah, pretty sure that's what happened from what I'm seeing.
My buildfarm critters seem to be OK with this, so pushed.
regards, tom lane