Thread: BUG #18839: ARMv7 builds fail due to missing __crc32cw and similar

The following bug has been logged on the website:

Bug reference:      18839
Logged by:          Mathew
Email address:      mat999@gmail.com
PostgreSQL version: 14.17
Operating system:   Debian
Description:

Changes made to support hardware CRC seem to have broken ARMv7 builds for
us.

When building we are now getting:

```
pg_crc32c_armv8.c:(.text+0x40): undefined reference to `__crc32cw'
/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x6c): undefined reference to
`__crc32cw'
/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x7a): undefined reference to
`__crc32ch'
/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x92): undefined reference to
`__crc32cd'
/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x8a): undefined reference to
`__crc32cb'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:66: postgres] Error 1
make[1]: *** [Makefile:42: all-backend-recurse] Error 2
make: *** [GNUmakefile:11: all-src-recurse] Error 2", "stderr_lines":
["pg_crc32c_armv8.c: In function ‘pg_comp_crc32c_armv8’:",
"pg_crc32c_armv8.c:35:23: warning: implicit declaration of function
‘__crc32cb’ [-Wimplicit-function-declaration]", "   35 |                 crc
= __crc32cb(crc, *p);", "      |                       ^~~~~~~~~",
"pg_crc32c_armv8.c:41:23: warning: implicit declaration of function
‘__crc32ch’ [-Wimplicit-function-declaration]", "   41 |                 crc
= __crc32ch(crc, *(uint16 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:47:23: warning: implicit declaration of
function ‘__crc32cw’ [-Wimplicit-function-declaration]", "   47 |
     crc = __crc32cw(crc, *(uint32 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:54:23: warning: implicit declaration of
function ‘__crc32cd’ [-Wimplicit-function-declaration]", "   54 |
     crc = __crc32cd(crc, *(uint64 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c: In function ‘pg_comp_crc32c_armv8’:",
"pg_crc32c_armv8.c:35:23: warning: implicit declaration of function
‘__crc32cb’ [-Wimplicit-function-declaration]", "   35 |                 crc
= __crc32cb(crc, *p);", "      |                       ^~~~~~~~~",
"pg_crc32c_armv8.c:41:23: warning: implicit declaration of function
‘__crc32ch’ [-Wimplicit-function-declaration]", "   41 |                 crc
= __crc32ch(crc, *(uint16 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:47:23: warning: implicit declaration of
function ‘__crc32cw’ [-Wimplicit-function-declaration]", "   47 |
     crc = __crc32cw(crc, *(uint32 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:54:23: warning: implicit declaration of
function ‘__crc32cd’ [-Wimplicit-function-declaration]", "   54 |
     crc = __crc32cd(crc, *(uint64 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c: In function ‘pg_comp_crc32c_armv8’:",
"pg_crc32c_armv8.c:35:23: warning: implicit declaration of function
‘__crc32cb’ [-Wimplicit-function-declaration]", "   35 |                 crc
= __crc32cb(crc, *p);", "      |                       ^~~~~~~~~",
"pg_crc32c_armv8.c:41:23: warning: implicit declaration of function
‘__crc32ch’ [-Wimplicit-function-declaration]", "   41 |                 crc
= __crc32ch(crc, *(uint16 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:47:23: warning: implicit declaration of
function ‘__crc32cw’ [-Wimplicit-function-declaration]", "   47 |
     crc = __crc32cw(crc, *(uint32 *) p);", "      |
^~~~~~~~~", "pg_crc32c_armv8.c:54:23: warning: implicit declaration of
function ‘__crc32cd’ [-Wimplicit-function-declaration]", "   54 |
     crc = __crc32cd(crc, *(uint64 *) p);", "      |
^~~~~~~~~", "/usr/bin/ld:
../../src/port/libpgport_srv.a(pg_crc32c_armv8_srv.o): in function
`pg_comp_crc32c_armv8':", "pg_crc32c_armv8.c:(.text+0x16): undefined
reference to `__crc32cb'", "/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x2a):
undefined reference to `__crc32ch'", "/usr/bin/ld:
pg_crc32c_armv8.c:(.text+0x40): undefined reference to `__crc32cw'",
"/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x6c): undefined reference to
`__crc32cw'", "/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x7a): undefined
reference to `__crc32ch'", "/usr/bin/ld: pg_crc32c_armv8.c:(.text+0x92):
undefined reference to `__crc32cd'", "/usr/bin/ld:
pg_crc32c_armv8.c:(.text+0x8a): undefined reference to `__crc32cb'",
"collect2: error: ld returned 1 exit status"
```

It seems like the configure test is passing when it shouldnt. This CPU does
not support hardware CRC32.

PG Bug reporting form <noreply@postgresql.org> writes:
> Changes made to support hardware CRC seem to have broken ARMv7 builds for
> us.

What is your build platform *exactly* -- what OS version, C compiler
version, etc?  "Debian" is not an adequate description.  Did you
use any nondefault configure options?  Which PG version did you
last build successfully?

            regards, tom lane

On Wed, Mar 12, 2025 at 10:06:41AM -0400, Tom Lane wrote:
> PG Bug reporting form <noreply@postgresql.org> writes:
>> Changes made to support hardware CRC seem to have broken ARMv7 builds for
>> us.
> 
> What is your build platform *exactly* -- what OS version, C compiler
> version, etc?  "Debian" is not an adequate description.  Did you
> use any nondefault configure options?  Which PG version did you
> last build successfully?

The relevant parts of config.log would also be helpful.  The only recent
change I see on REL_14_STABLE is commit 2fc0199.

-- 
nathan

Mathew Heard <mat999@gmail.com> writes:
> The relevant commit we have removed from our build that has returned the
> ability to build is.
> https://github.com/postgres/postgres/commit/5980f1884fc911af120c98ad440b9546ed9012c5

> GCC 12.2.0-3 (latest for debian bullseye)
> Debian Bullseye
> ARMv7 (Allwinner H3)

We have buildfarm animals matching that description, eg grison:

https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=grison&br=master

So this is not enough info to debug why it doesn't work for you.

            regards, tom lane

Mathew Heard <mat999@gmail.com> writes:
> The relevant commit we have removed from our build that has returned the
> ability to build is.
> https://github.com/postgres/postgres/commit/5980f1884fc911af120c98ad440b9546ed9012c5
> GCC 12.2.0-3 (latest for debian bullseye)
> Debian Bullseye
> ARMv7 (Allwinner H3)

I spent some time today trying to reproduce this, without success.
One thing I noticed is that Debian 11 seems to have shipped with
gcc 10.2.1, while gcc 12.2.0 comes with Debian 12 (Bookworm).
I don't doubt that you can get gcc 12 for Debian 11, but I wonder
if there's some OS-vs-compiler version skew underlying this issue
for you.

The real question though is how the configure test could succeed
only to get a failure in the main build.  If you would get
"undefined reference to `__crc32cb'" etc in the main build,
the configure probe should have failed in the same way.
I tried assorted -march settings with both of those gcc versions
and it acted as-expected for me.

            regards, tom lane

Mathew Heard <mat999@gmail.com> writes:
> If I am correct, postgresql is identifying the CPU of the build system as
> ARMv8 in that test, and using this rather than the target architecture.

The configure script isn't "identifying" anything.  It's just seeing
whether references to __crc32cb() etc will compile with different
-march flags.  It's not apparent why a successful test of that sort
would not lead to a successful compilation with the same flags later
on.  We might do a *run time* probe of the actual CPU type, but that
has nothing to do with compilation.

You still haven't told us which -march setting configure is selecting,
nor provided the config.log trace showing why it chose that one.
You also haven't described the compilation environment in any detail
--- eg, I still am not sure if this is a hard-float or soft-float
environment.

            regards, tom lane

I wrote:
> The configure script isn't "identifying" anything.  It's just seeing
> whether references to __crc32cb() etc will compile with different
> -march flags.  It's not apparent why a successful test of that sort
> would not lead to a successful compilation with the same flags later
> on.

Actually ... looking harder at the test code, perhaps I see a way.
We're testing this:

   unsigned int crc = 0;
   crc = __crc32cb(crc, 0);
   crc = __crc32ch(crc, 0);
   crc = __crc32cw(crc, 0);
   crc = __crc32cd(crc, 0);
   /* return computed value, to prevent the above being optimized away */
   return crc == 0;

but that "prevent the above being optimized away" looks mighty leaky.
Specifically, there's nothing stopping the compiler from folding all
these CRC calls to constants, since the input is constant.  So in
an environment where the compiler knows these functions but glibc
doesn't have them, perhaps we could get a false pass --- and then
the real code with non-constant inputs could fail?

I'm not very convinced of this theory, because I checked with gcc
12.2.0 and it wouldn't do that even at max -O level.  But maybe
your compiler is different.  Could you try the attached patch
for configure and see if it arrives at the right conclusions?

            regards, tom lane

diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index d3562d6feee..a4b005d07ca 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -659,9 +659,9 @@ AC_DEFUN([PGAC_ARMV8_CRC32C_INTRINSICS],
 AC_CACHE_CHECK([for __crc32cb, __crc32ch, __crc32cw, and __crc32cd with CFLAGS=$1], [Ac_cachevar],
 [pgac_save_CFLAGS=$CFLAGS
 CFLAGS="$pgac_save_CFLAGS $1"
-AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_acle.h>],
-  [unsigned int crc = 0;
-   crc = __crc32cb(crc, 0);
+AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_acle.h>
+unsigned int crc;],
+  [crc = __crc32cb(crc, 0);
    crc = __crc32ch(crc, 0);
    crc = __crc32cw(crc, 0);
    crc = __crc32cd(crc, 0);
diff --git a/configure b/configure
index fa9085b1c0d..3f89837e031 100755
--- a/configure
+++ b/configure
@@ -18741,11 +18741,11 @@ CFLAGS="$pgac_save_CFLAGS "
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 #include <arm_acle.h>
+unsigned int crc;
 int
 main ()
 {
-unsigned int crc = 0;
-   crc = __crc32cb(crc, 0);
+crc = __crc32cb(crc, 0);
    crc = __crc32ch(crc, 0);
    crc = __crc32cw(crc, 0);
    crc = __crc32cd(crc, 0);
@@ -18782,11 +18782,11 @@ CFLAGS="$pgac_save_CFLAGS -march=armv8-a+crc+simd"
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 #include <arm_acle.h>
+unsigned int crc;
 int
 main ()
 {
-unsigned int crc = 0;
-   crc = __crc32cb(crc, 0);
+crc = __crc32cb(crc, 0);
    crc = __crc32ch(crc, 0);
    crc = __crc32cw(crc, 0);
    crc = __crc32cd(crc, 0);
@@ -18823,11 +18823,11 @@ CFLAGS="$pgac_save_CFLAGS -march=armv8-a+crc"
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 #include <arm_acle.h>
+unsigned int crc;
 int
 main ()
 {
-unsigned int crc = 0;
-   crc = __crc32cb(crc, 0);
+crc = __crc32cb(crc, 0);
    crc = __crc32ch(crc, 0);
    crc = __crc32cw(crc, 0);
    crc = __crc32cd(crc, 0);

Mathew Heard <mat999@gmail.com> writes:
> When the configure test is running its forcing armv8-a+crc+simd (has
> __crc32cb, etc)  rather than testing for the feature against target
> architecture (no __crc32cb, etc). While make is compiling for the target
> architecture armv7ve+simd (no __crc32cb, etc). During the make it is of
> course failing to find __crc32cb etc, they only existed during the
> configure test due to the custom cflags used for that feature test (march
> armv8-a+crc+simd).

No, because when pg_crc32c_armv8.c is compiled, that same -march
setting is used for that one specific file -- trace through the
use of CFLAGS_CRC in the Makefiles.  For instance, on buildfarm
member grison[1] which seems like it should be a pretty close match
for your situation, the configure log shows

checking for __crc32cb, __crc32ch, __crc32cw, and __crc32cd with CFLAGS=... (cached) no
checking for __crc32cb, __crc32ch, __crc32cw, and __crc32cd with CFLAGS=-march=armv8-a+crc+simd... (cached) yes
checking which CRC-32C implementation to use... ARMv8 CRC instructions with runtime check

and then in the build log we find

ccache gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute-Wimplicit-fallthrough=3 -Wcast-function-type -Wshadow=compatible-local -Wformat-security
-fno-strict-aliasing-fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -O2
-march=armv8-a+crc+simd-I../../src/port -DFRONTEND -I../../src/include  -D_GNU_SOURCE -I/usr/include/libxml2   -c -o
pg_crc32c_armv8.opg_crc32c_armv8.c 

(note the -march flag) so that's expected to compile, and does,
even though the platform is armv7l in general.  I have no idea whether
grison's underlying hardware is v7 or v8, and it shouldn't matter for
this purpose, because the only thing at stake is the build toolchain's
behavior.

> I'm not really sure what's so confusing about this situation. It seems like
> a configure feature test that will always detect CRC support on any ARM gcc
> (that supports those functions). I'm mostly surprised your arm build
> environment is passing.

I'm not sure how I can put this any more plainly: it works fine for
everyone but you, including ARMv7 machines.  I have no intention of
regressing performance for everyone who runs on mixed ARMv7/v8
platforms in order to un-break your build.  We need to figure out what
is unusual about your build environment that is causing this mechanism
not to work as-expected.

            regards, tom lane

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grison&dt=2025-03-14%2000%3A04%3A12