Thread: Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
This seems to work:
# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
# Fails with 17.0 succeeds with 16.4
ARG PG=17.0
USER root
RUN apk update && apk add --no-cache --update-cache \
ca-certificates-bundle \
util-linux-dev \
execline-dev \
libedit-dev \
libxml2-dev \
clang17-dev \
llvm18-dev \
build-base \
net-tools \
zlib-dev \
autoconf \
automake \
busybox \
icu-dev \
clang17 \
llvm18 \
cmake \
bison \
meson \
ninja \
flex \
perl \
curl \
bash
WORKDIR /app
RUN curl -L https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz > openssl.tar.gz
RUN mkdir openssl && tar --extract --file=openssl.tar.gz --strip-components=1 --directory=openssl && rm openssl.tar.gz
RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
linux-$(uname -m) \
--prefix=/usr \
--libdir=lib \
--openssldir=/etc/ssl \
enable-ktls \
shared \
no-zlib \
no-async \
no-comp \
no-idea \
no-mdc2 \
no-rc5 \
no-ec2m \
no-sm2 \
no-sm4 \
no-ssl3 \
no-seed \
no-weak-ssl-ciphers \
-Wa,--noexecstack && \
perl configdata.pm --dump && \
make -j$(nproc) && make install
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
COPY <<EOF ./pg.patch
Subject: [PATCH] Fix static build
---
Index: src/bin/initdb/Makefile
<+>UTF-8
===================================================================
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
--- a/src/bin/initdb/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/bin/initdb/Makefile (date 1728466911177)
@@ -20,7 +20,8 @@
# from libpq, else we have risks of version skew if we run with a libpq
# shared library from a different PG version. Define
# USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
-override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
# We need libpq only because fe_utils does.
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(ICU_LIBS)
Index: src/common/Makefile
<+>UTF-8
===================================================================
diff --git a/src/common/Makefile b/src/common/Makefile
--- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/common/Makefile (date 1728466354406)
@@ -143,7 +143,7 @@
# Files in libpgcommon.a should use/export the "xxx_private" versions
# of pg_char_to_encoding() and friends.
#
-$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
+#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
#
EOF
RUN apk update && apk add --no-cache --update-cache git
RUN cd postgresql && git apply ../pg.patch
RUN cd postgresql && export CC=clang && export CXX=clang++ && \
./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local && \
cd src/include && make && make install && \
cd ../common && make && make install && \
cd ../port && make && make install && \
cd ../interfaces/libpq && make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
# Fails with "-static" succeeds without
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []
On Wed, Oct 9, 2024 at 10:10 AM Mikael Sand <msand@seaber.io> wrote:
Hello dear HackersI'm trying to upgrade to v17 and encountering a strange issue.I've made a minimal reproduction that works with 16.4 but fails with 17.0, it also works without the "-static" compile flag:# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
# Fails with 17.0 succeeds with 16.4
ARG PG=17.0
USER root
RUN apk update && apk add --no-cache --update-cache \
ca-certificates-bundle \
util-linux-dev \
clang17-dev \
execline-dev \
llvm18-dev \
libedit-dev \
libxml2-dev \
build-base \
net-tools \
clang17 \
zlib-dev \
autoconf \
automake \
busybox \
llvm18 \
icu-dev \
cmake \
bison \
flex \
perl \
curl \
bash \
flex
WORKDIR /app
RUN curl -L https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz > openssl.tar.gz
RUN mkdir openssl && tar --extract --file=openssl.tar.gz --strip-components=1 --directory=openssl && rm openssl.tar.gz
RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
linux-$(uname -m) \
--prefix=/usr \
--libdir=lib \
--openssldir=/etc/ssl \
enable-ktls \
shared \
no-zlib \
no-async \
no-comp \
no-idea \
no-mdc2 \
no-rc5 \
no-ec2m \
no-sm2 \
no-sm4 \
no-ssl3 \
no-seed \
no-weak-ssl-ciphers \
-Wa,--noexecstack && \
perl configdata.pm --dump && \
make -j$(nproc) && make install
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
RUN cd postgresql && CC=clang CXX=clang++ ./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local
RUN cd postgresql/src/include && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/common && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/port && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/interfaces/libpq && CC=clang CXX=clang++ make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
# Fails with "-static" succeeds without
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []And here is the output:15/15RUN CC=clang CXX=clang++ clang++ -std=c++20 -static -flto -O3 -march=native -Wpedantic -Wall -Wextra -Wsign-conversion -Wconversion -o main main.cpp -lpq -lpgcommon -lpgport -lssl -lcrypto -lpthread -ldlERROR1.0s1/usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `pqConnectOptions2':2fe-connect.c:(.text+0x1cb4): undefined reference to `pg_encoding_to_char'3/usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `PQsetClientEncoding':4fe-connect.c:(.text+0x64a8): undefined reference to `pg_encoding_to_char'5/usr/bin/ld: /usr/local/lib/libpq.a(fe-exec.o): in function `pqSaveParameterStatus':6fe-exec.c:(.text+0x1168): undefined reference to `pg_char_to_encoding'7/usr/bin/ld: /usr/local/lib/libpq.a(fe-misc.o): in function `PQenv2encoding':8fe-misc.c:(.text+0x1394): undefined reference to `pg_char_to_encoding'9clang++: error: linker command failed with exit code 1 (use -v to see invocation)Any ideas?Best regardMikael Sand
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Here is a simpler Dockerfile for making a static build of a c++ application using libpq from postgresql 17
# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
e2fsprogs-dev \
readline-dev \
libxml2-dev \
openssl-dev \
zlib-dev \
clang17 \
icu-dev \
bison \
flex \
perl \
make \
curl \
git
ARG PG=17.0
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
COPY <<EOF ./pg.patch
Subject: [PATCH] Fix static build
---
Index: src/bin/initdb/Makefile
<+>UTF-8
===================================================================
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
--- a/src/bin/initdb/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/bin/initdb/Makefile (date 1728466911177)
@@ -20,7 +20,8 @@
# from libpq, else we have risks of version skew if we run with a libpq
# shared library from a different PG version. Define
# USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
-override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
# We need libpq only because fe_utils does.
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(ICU_LIBS)
Index: src/common/Makefile
<+>UTF-8
===================================================================
diff --git a/src/common/Makefile b/src/common/Makefile
--- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/common/Makefile (date 1728466354406)
@@ -143,7 +143,7 @@
# Files in libpgcommon.a should use/export the "xxx_private" versions
# of pg_char_to_encoding() and friends.
#
-$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
+#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
#
EOF
RUN cd postgresql && git apply ../pg.patch
RUN cd postgresql && export CC=clang && export CXX=clang++ && \
./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local && \
cd src/include && make && make install && \
cd ../common && make && make install && \
cd ../port && make && make install && \
cd ../interfaces/libpq && make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []
And another to verify that the build is actually static, run inside wolfi undistro made for running a single static binary (and using clang-19 instead of 17):
# syntax=docker/dockerfile:1
FROM cgr.dev/chainguard/git:latest-dev AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
e2fsprogs-dev \
clang-19-dev \
execline-dev \
libedit-dev \
libxml2-dev \
openssl-dev \
llvm-19-dev \
build-base \
clang-19 \
zlib-dev \
icu-dev \
llvm-19 \
bison \
flex \
perl \
curl
ARG PG=17.0
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
COPY <<EOF ./pg.patch
Subject: [PATCH] Fix static build
---
Index: src/bin/initdb/Makefile
<+>UTF-8
===================================================================
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
--- a/src/bin/initdb/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/bin/initdb/Makefile (date 1728466911177)
@@ -20,7 +20,8 @@
# from libpq, else we have risks of version skew if we run with a libpq
# shared library from a different PG version. Define
# USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
-override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
# We need libpq only because fe_utils does.
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(ICU_LIBS)
Index: src/common/Makefile
<+>UTF-8
===================================================================
diff --git a/src/common/Makefile b/src/common/Makefile
--- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/common/Makefile (date 1728466354406)
@@ -143,7 +143,7 @@
# Files in libpgcommon.a should use/export the "xxx_private" versions
# of pg_char_to_encoding() and friends.
#
-$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
+#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
#
EOF
RUN cd postgresql && git apply ../pg.patch
RUN cd postgresql && export CC=clang && export CXX=clang++ && \
./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local && \
cd src/include && make && make install && \
cd ../common && make && make install && \
cd ../port && make && make install && \
cd ../interfaces/libpq && make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
RUN ln /usr/lib/llvm-19/lib/LLVMgold.so /usr/lib/LLVMgold.so
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []
FROM cgr.dev/chainguard/static:latest AS runner
WORKDIR /app
COPY --from=builder --chown=65532:65532 /app/main .
USER 65532
ENTRYPOINT ["./main"]
CMD []
Hope this can help somebody
Br Mikael
On Wed, Oct 9, 2024 at 11:51 AM Mikael Sand <msand@seaber.io> wrote:
This seems to work:# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
# Fails with 17.0 succeeds with 16.4
ARG PG=17.0
USER root
RUN apk update && apk add --no-cache --update-cache \
ca-certificates-bundle \
util-linux-dev \
execline-dev \
libedit-dev \
libxml2-dev \
clang17-dev \
llvm18-dev \
build-base \
net-tools \
zlib-dev \
autoconf \
automake \
busybox \
icu-dev \
clang17 \
llvm18 \
cmake \
bison \
meson \
ninja \
flex \
perl \
curl \
bash
WORKDIR /app
RUN curl -L https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz > openssl.tar.gz
RUN mkdir openssl && tar --extract --file=openssl.tar.gz --strip-components=1 --directory=openssl && rm openssl.tar.gz
RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
linux-$(uname -m) \
--prefix=/usr \
--libdir=lib \
--openssldir=/etc/ssl \
enable-ktls \
shared \
no-zlib \
no-async \
no-comp \
no-idea \
no-mdc2 \
no-rc5 \
no-ec2m \
no-sm2 \
no-sm4 \
no-ssl3 \
no-seed \
no-weak-ssl-ciphers \
-Wa,--noexecstack && \
perl configdata.pm --dump && \
make -j$(nproc) && make install
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
COPY <<EOF ./pg.patch
Subject: [PATCH] Fix static build
---
Index: src/bin/initdb/Makefile
<+>UTF-8
===================================================================
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
--- a/src/bin/initdb/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/bin/initdb/Makefile (date 1728466911177)
@@ -20,7 +20,8 @@
# from libpq, else we have risks of version skew if we run with a libpq
# shared library from a different PG version. Define
# USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
-override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
# We need libpq only because fe_utils does.
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(ICU_LIBS)
Index: src/common/Makefile
<+>UTF-8
===================================================================
diff --git a/src/common/Makefile b/src/common/Makefile
--- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/common/Makefile (date 1728466354406)
@@ -143,7 +143,7 @@
# Files in libpgcommon.a should use/export the "xxx_private" versions
# of pg_char_to_encoding() and friends.
#
-$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
+#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
#
EOF
RUN apk update && apk add --no-cache --update-cache git
RUN cd postgresql && git apply ../pg.patch
RUN cd postgresql && export CC=clang && export CXX=clang++ && \
./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local && \
cd src/include && make && make install && \
cd ../common && make && make install && \
cd ../port && make && make install && \
cd ../interfaces/libpq && make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
# Fails with "-static" succeeds without
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []On Wed, Oct 9, 2024 at 10:10 AM Mikael Sand <msand@seaber.io> wrote:Hello dear HackersI'm trying to upgrade to v17 and encountering a strange issue.I've made a minimal reproduction that works with 16.4 but fails with 17.0, it also works without the "-static" compile flag:# syntax=docker/dockerfile:1
FROM alpine:3.20 AS builder
# Fails with 17.0 succeeds with 16.4
ARG PG=17.0
USER root
RUN apk update && apk add --no-cache --update-cache \
ca-certificates-bundle \
util-linux-dev \
clang17-dev \
execline-dev \
llvm18-dev \
libedit-dev \
libxml2-dev \
build-base \
net-tools \
clang17 \
zlib-dev \
autoconf \
automake \
busybox \
llvm18 \
icu-dev \
cmake \
bison \
flex \
perl \
curl \
bash \
flex
WORKDIR /app
RUN curl -L https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz > openssl.tar.gz
RUN mkdir openssl && tar --extract --file=openssl.tar.gz --strip-components=1 --directory=openssl && rm openssl.tar.gz
RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
linux-$(uname -m) \
--prefix=/usr \
--libdir=lib \
--openssldir=/etc/ssl \
enable-ktls \
shared \
no-zlib \
no-async \
no-comp \
no-idea \
no-mdc2 \
no-rc5 \
no-ec2m \
no-sm2 \
no-sm4 \
no-ssl3 \
no-seed \
no-weak-ssl-ciphers \
-Wa,--noexecstack && \
perl configdata.pm --dump && \
make -j$(nproc) && make install
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 > postgresql.tar.bz2
RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2 --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
RUN cd postgresql && CC=clang CXX=clang++ ./configure --with-openssl --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local
RUN cd postgresql/src/include && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/common && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/port && CC=clang CXX=clang++ make && make install
RUN cd postgresql/src/interfaces/libpq && CC=clang CXX=clang++ make && make install
COPY <<EOF ./main.cpp
#include <algorithm>
#include <iostream>
#include <chrono>
#include <thread>
#include <cstring>
#include <csignal>
#include <vector>
#include <libpq-fe.h>
void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
raise(info->si_signo);
_exit(EXIT_FAILURE);
}
void registerSignalHandlers() {
std::vector<int> signals = {
// Signals for which the default action is "Core".
SIGABRT, // Abort signal from abort(3)
SIGBUS, // Bus error (bad memory access)
SIGFPE, // Floating point exception
SIGILL, // Illegal Instruction
SIGIOT, // IOT trap. A synonym for SIGABRT
SIGQUIT, // Quit from keyboard
SIGSEGV, // Invalid memory reference
SIGSYS, // Bad argument to routine (SVr4)
SIGTRAP, // Trace/breakpoint trap
SIGXCPU, // CPU time limit exceeded (4.2BSD)
SIGXFSZ, // File size limit exceeded (4.2BSD)
SIGTERM
};
for (size_t i = 0; i < signals.size(); ++i) {
struct sigaction action;
memset(&action, 0, sizeof action);
action.sa_flags = static_cast<int>(SA_SIGINFO | SA_ONSTACK | SA_NODEFER | SA_RESETHAND);
sigfillset(&action.sa_mask);
sigdelset(&action.sa_mask, signals[i]);
action.sa_sigaction = &sig_handler;
sigaction(signals[i], &action, nullptr);
}
}
int main() {
registerSignalHandlers();
std::cout << "start" << std::endl;
PGconn *conn = PQconnectdb("");
if (conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
# Fails with "-static" succeeds without
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []And here is the output:15/15RUN CC=clang CXX=clang++ clang++ -std=c++20 -static -flto -O3 -march=native -Wpedantic -Wall -Wextra -Wsign-conversion -Wconversion -o main main.cpp -lpq -lpgcommon -lpgport -lssl -lcrypto -lpthread -ldlERROR1.0s1/usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `pqConnectOptions2':2fe-connect.c:(.text+0x1cb4): undefined reference to `pg_encoding_to_char'3/usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `PQsetClientEncoding':4fe-connect.c:(.text+0x64a8): undefined reference to `pg_encoding_to_char'5/usr/bin/ld: /usr/local/lib/libpq.a(fe-exec.o): in function `pqSaveParameterStatus':6fe-exec.c:(.text+0x1168): undefined reference to `pg_char_to_encoding'7/usr/bin/ld: /usr/local/lib/libpq.a(fe-misc.o): in function `PQenv2encoding':8fe-misc.c:(.text+0x1394): undefined reference to `pg_char_to_encoding'9clang++: error: linker command failed with exit code 1 (use -v to see invocation)Any ideas?Best regardMikael Sand
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Sorry for not having a properly minimal reproduction to begin with, seems I can't reduce this any further at the moment at least:
# syntax=docker/dockerfile:1
FROM chainguard/git:latest-dev AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
clang-19 \
bison \
curl \
flex \
make \
perl
ARG PG=17.0
RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 | tar -xj
COPY <<EOF ./pg.patch
Subject: [PATCH] Fix static build
---
Index: src/bin/initdb/Makefile
<+>UTF-8
===================================================================
diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
--- a/src/bin/initdb/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/bin/initdb/Makefile (date 1728466911177)
@@ -20,7 +20,8 @@
# from libpq, else we have risks of version skew if we run with a libpq
# shared library from a different PG version. Define
# USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
-override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
+override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
# We need libpq only because fe_utils does.
LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) $(ICU_LIBS)
Index: src/common/Makefile
<+>UTF-8
===================================================================
diff --git a/src/common/Makefile b/src/common/Makefile
--- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
+++ b/src/common/Makefile (date 1728466354406)
@@ -143,7 +143,7 @@
# Files in libpgcommon.a should use/export the "xxx_private" versions
# of pg_char_to_encoding() and friends.
#
-$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
+#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
#
EOF
RUN cd postgresql-$PG && \
git apply ../pg.patch && \
export CC=clang CXX=clang++ && \
./configure --without-openssl --without-icu --without-ldap --without-readline --without-zlib --without-tcl --without-python --prefix=/usr/local && \
cd src && for lib in include common port interfaces/libpq ; do cd $lib && make && make install && cd - ; done
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
RUN clang++ -static -o main main.cpp -lpq -lpgcommon -lpgport
ENTRYPOINT ["./main"]
CMD []
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Aleksander Alekseev
Date:
Hi Mikael, > Sorry for not having a properly minimal reproduction to begin with, seems I can't reduce this any further at the momentat least: > [...] Is there any particular reason for using pg.patch in your Dockerfile or not using well regarded / official [1][2] Docker images [3] ? It seems to me that basically you are complaining that you patched Postgres and it didn't compile after that. [1]: I don't use Docker a lot at present and I'm not sure how oficial these images really are and/or who is maintaining them. [2]: see also https://postgr.es/m/05d86238a6e46da637c415bd261451de%40postgresql.org [3]: https://hub.docker.com/_/postgres -- Best regards, Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Aleksander Alekseev
Date:
Hi Mikael, Please use the "Reply to All" button - send your emails to the mailing list, not to me directly. > The official packages and the original source tar both fail to compile, the patch enables it to compile. I just run: ``` docker run -e POSTGRES_PASSWORD=foo postgres:17.0-alpine3.20 ``` ... on my spare Raspberry Pi and it works just fine. Also PostgreSQL is tested on Alpine [1]. It means that your custom Dockerfile is wrong. I suggest comparing it with the one published on hub.docker.com [2]. Also I would recommend reading the documentation section about compiling PostgreSQL from source code [3]. I have a few scripts [4] that may help you correct the Dockerfile as well [4]. All this of course assumes that you really want your custom Dockerfile instead of using "official" / well regarded 17.0-alpine3.20 one. [1]: https://buildfarm.postgresql.org/cgi-bin/show_status.pl [2]: https://github.com/docker-library/postgres/blob/172544062d1031004b241e917f5f3f9dfebc0df5/17/alpine3.20/Dockerfile [3]: https://www.postgresql.org/docs/current/installation.html [4]: https://github.com/afiskon/pgscripts/ -- Best regards, Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Hello Aleksander
Br Mikael
On Thu, Oct 10, 2024 at 12:16 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Hi Mikael,
Please use the "Reply to All" button - send your emails to the mailing
list, not to me directly.
> The official packages and the original source tar both fail to compile, the patch enables it to compile.
I just run:
```
docker run -e POSTGRES_PASSWORD=foo postgres:17.0-alpine3.20
```
... on my spare Raspberry Pi and it works just fine. Also PostgreSQL
is tested on Alpine [1].
It means that your custom Dockerfile is wrong. I suggest comparing it
with the one published on hub.docker.com [2]. Also I would recommend
reading the documentation section about compiling PostgreSQL from
source code [3]. I have a few scripts [4] that may help you correct
the Dockerfile as well [4].
All this of course assumes that you really want your custom Dockerfile
instead of using "official" / well regarded 17.0-alpine3.20 one.
[1]: https://buildfarm.postgresql.org/cgi-bin/show_status.pl
[2]: https://github.com/docker-library/postgres/blob/172544062d1031004b241e917f5f3f9dfebc0df5/17/alpine3.20/Dockerfile
[3]: https://www.postgresql.org/docs/current/installation.html
[4]: https://github.com/afiskon/pgscripts/
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
E.g. this fails to compile:
FROM chainguard/git:latest-dev AS builder
USER root
RUN apk update && apk add --no-cache --update-cache clang-19 postgresql-17-dev openssl-dev
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
RUN clang++ -static -o main main.cpp -lpq -lpgcommon -lpgport -lssl -lcrypto
On Thu, Oct 10, 2024 at 12:16 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Hi Mikael,
Please use the "Reply to All" button - send your emails to the mailing
list, not to me directly.
> The official packages and the original source tar both fail to compile, the patch enables it to compile.
I just run:
```
docker run -e POSTGRES_PASSWORD=foo postgres:17.0-alpine3.20
```
... on my spare Raspberry Pi and it works just fine. Also PostgreSQL
is tested on Alpine [1].
It means that your custom Dockerfile is wrong. I suggest comparing it
with the one published on hub.docker.com [2]. Also I would recommend
reading the documentation section about compiling PostgreSQL from
source code [3]. I have a few scripts [4] that may help you correct
the Dockerfile as well [4].
All this of course assumes that you really want your custom Dockerfile
instead of using "official" / well regarded 17.0-alpine3.20 one.
[1]: https://buildfarm.postgresql.org/cgi-bin/show_status.pl
[2]: https://github.com/docker-library/postgres/blob/172544062d1031004b241e917f5f3f9dfebc0df5/17/alpine3.20/Dockerfile
[3]: https://www.postgresql.org/docs/current/installation.html
[4]: https://github.com/afiskon/pgscripts/
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Aleksander Alekseev
Date:
Hi Mikael, > This is for compiling a c++ application that uses libpq with the -static flag, the server compiles fine. OK. I couldn't quite do this because the only Linux machine I have at the moment runs Raspbian and there doesn't seem to be a static glibc available for it. But here is how to achieve what you want *in theory*. First compile Postgres from the source code, for instance (change the flags as needed, you probably want a release build): ``` # sudo apt install clang-16 # git clean -dfx CFLAGS="-static" meson setup --buildtype debug -Dicu=disabled -Dldap=disabled -Dreadline=disabled -Dzlib=disabled -Dlz4=disabled -Dprefix=/home/eax/pginstall build ninja -C build ``` I recommend using Meson and Ninja. Autotools and Make are still supported but are slow and probably will be gone in a few years. The "ninja -C build" steps fail for me with various errors about the need for static glibc but I think it should work on other distributions and architectures. Next install Postgres. I use a script for this [1]: ``` ~/pgscripts/single-install-meson.sh ``` Change the script as needed - you probably don't need pg_ctl, createdb, etc for your task. Now in order to compile and execute your C++ program: ``` export PRFX=/home/eax/pginstall g++ -g -Wall -o test_libpq -I$PRFX/include -L$PRFX/lib/aarch64-linux-gnu/ test_libpq.cpp -lpq -static LD_LIBRARY_PATH=$PRFX/lib/aarch64-linux-gnu/ ./test_libpq ``` This being said I don't recall seeing anything about the support of static linking of libpq in the documentation [2]. To my knowledge this is not officially supported / tested / maintained which means you and your colleagues are on your own with the -static flag. Since you are already using Docker, perhaps the easiest thing to do would be to back the application and all its dependencies (dynamically linked) in a Docker container. Good luck. [1]: https://github.com/afiskon/pgscripts/ [2]: https://www.postgresql.org/docs/current/libpq.html -- Best regards, Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Hmm, so is static linking of all applications that include libpq intentionally broken?
At least googling around, it seems like a relatively common practice.
At least it seems that 2ndquadrant builds and uses libpq statically.
https://www.postgresql.org/message-id/20327.1501536978%40sss.pgh.pa.us
https://www.postgresql.org/message-id/20327.1501536978%40sss.pgh.pa.us
I don't see how your script would fix the naming of these functions, can you elaborate?
Here is a Dockerfile demonstrating the same issue with the postgres:17.0-alpine3.20 image:
FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
cyrus-sasl-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
zstd-static \
zlib-static \
libedit-dev \
openssl-dev \
lz4-static \
e2fsprogs \
zstd-dev \
keyutils \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
RUN curl -L https://kerberos.org/dist/krb5/1.21/krb5-1.21.3.tar.gz > krb5-1.21.3.tar.gz && tar xf krb5-1.21.3.tar.gz
RUN cd krb5-1.21.3/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
RUN curl -L https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.28/cyrus-sasl-2.1.28.tar.gz > cyrus-sasl-2.1.28.tar.gz
RUN tar xf cyrus-sasl-2.1.28.tar.gz && cd cyrus-sasl-2.1.28 && ./configure --enable-static && make && make install
RUN clang++ -fno-common -static -o main main.cpp \
-L/usr/local/lib -lpq -L/usr/lib/llvm15/lib -L/usr/local/lib -lpgcommon -lpgport -lgssapi_krb5 -lm -lldap -lcrypto -ldl -pthread \
-lldap -levent -lsasl2 -lssl -lcrypto -llber -levent \
-lsqlite3 \
-L/usr/local/lib -lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm \
-lssl \
-lgssapi_krb5
On Thu, Oct 10, 2024 at 1:51 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Hi Mikael,
> This is for compiling a c++ application that uses libpq with the -static flag, the server compiles fine.
OK. I couldn't quite do this because the only Linux machine I have at
the moment runs Raspbian and there doesn't seem to be a static glibc
available for it. But here is how to achieve what you want *in
theory*.
First compile Postgres from the source code, for instance (change the
flags as needed, you probably want a release build):
```
# sudo apt install clang-16
# git clean -dfx
CFLAGS="-static" meson setup --buildtype debug -Dicu=disabled
-Dldap=disabled -Dreadline=disabled -Dzlib=disabled -Dlz4=disabled
-Dprefix=/home/eax/pginstall build
ninja -C build
```
I recommend using Meson and Ninja. Autotools and Make are still
supported but are slow and probably will be gone in a few years. The
"ninja -C build" steps fail for me with various errors about the need
for static glibc but I think it should work on other distributions and
architectures.
Next install Postgres. I use a script for this [1]:
```
~/pgscripts/single-install-meson.sh
```
Change the script as needed - you probably don't need pg_ctl,
createdb, etc for your task.
Now in order to compile and execute your C++ program:
```
export PRFX=/home/eax/pginstall
g++ -g -Wall -o test_libpq -I$PRFX/include
-L$PRFX/lib/aarch64-linux-gnu/ test_libpq.cpp -lpq -static
LD_LIBRARY_PATH=$PRFX/lib/aarch64-linux-gnu/ ./test_libpq
```
This being said I don't recall seeing anything about the support of
static linking of libpq in the documentation [2]. To my knowledge this
is not officially supported / tested / maintained which means you and
your colleagues are on your own with the -static flag. Since you are
already using Docker, perhaps the easiest thing to do would be to back
the application and all its dependencies (dynamically linked) in a
Docker container.
Good luck.
[1]: https://github.com/afiskon/pgscripts/
[2]: https://www.postgresql.org/docs/current/libpq.html
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
This seems to be the commit that caused this: https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545a
So it seems the commit is related to static linking.
Is it somehow possible to statically link in a version of pgcommon with correct function names into libpq?
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
I don't mind having this patch in use too much, I have a functional build and nothing to worry about, no luck necessary :)
But, I expect a lot of unnecessary churn in the community if this is not fixed properly.
Not all applications run inside docker, and not all can use dynamic linking.
People can dislike static linking for whatever reason, but that doesn't remove the need for it (or both) in some cases and the desire for it in others.
Judging from the commit, it doesn't seem like static linking is intentionally broken.
Should a test be added so as not to break this by mistake?
I must say I don't fully comprehend the context of that commit.
On Thu, Oct 10, 2024 at 5:06 PM Mikael Sand <msand@seaber.io> wrote:
This seems to be the commit that caused this: https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545aSo it seems the commit is related to static linking.Is it somehow possible to statically link in a version of pgcommon with correct function names into libpq?
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
E.g. this works with 16.4 but fails in 17.0:
FROM postgres:16.4-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
cyrus-sasl-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
zstd-static \
zlib-static \
libedit-dev \
openssl-dev \
lz4-static \
e2fsprogs \
zstd-dev \
keyutils \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
RUN curl -L https://kerberos.org/dist/krb5/1.21/krb5-1.21.3.tar.gz > krb5-1.21.3.tar.gz && tar xf krb5-1.21.3.tar.gz
RUN cd krb5-1.21.3/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
RUN curl -L https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.28/cyrus-sasl-2.1.28.tar.gz > cyrus-sasl-2.1.28.tar.gz
RUN tar xf cyrus-sasl-2.1.28.tar.gz && cd cyrus-sasl-2.1.28 && ./configure --enable-static && make && make install
RUN clang++ -fno-common -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon -lpgport \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
On Thu, Oct 10, 2024 at 5:28 PM Mikael Sand <msand@seaber.io> wrote:
I don't mind having this patch in use too much, I have a functional build and nothing to worry about, no luck necessary :)But, I expect a lot of unnecessary churn in the community if this is not fixed properly.Not all applications run inside docker, and not all can use dynamic linking.People can dislike static linking for whatever reason, but that doesn't remove the need for it (or both) in some cases and the desire for it in others.Judging from the commit, it doesn't seem like static linking is intentionally broken.Should a test be added so as not to break this by mistake?I must say I don't fully comprehend the context of that commit.On Thu, Oct 10, 2024 at 5:06 PM Mikael Sand <msand@seaber.io> wrote:This seems to be the commit that caused this: https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545aSo it seems the commit is related to static linking.Is it somehow possible to statically link in a version of pgcommon with correct function names into libpq?
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Tom Lane
Date:
Mikael Sand <msand@seaber.io> writes: > RUN clang++ -fno-common -static -o main main.cpp \ > -L/usr/local/lib -lpq -lpgcommon -lpgport \ > -lldap -lsasl2 -lssl -lcrypto -llber \ > -lgssapi_krb5 \ > -lkrb5 -lk5crypto -lcom_err -lkrb5support \ > -lgdbm The short answer here is that your link recipe is wrong, and has been wrong right along, though you accidentally got away with it before. The modules within libpq expect to be linked with libpgcommon_shlib and libpgport_shlib, not libpgcommon/libpgport. Having external code that needs to know explicitly about every one of a library's dependencies is one of many reasons why we discourage static linking. regards, tom lane
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Aleksander Alekseev
Date:
Mikael, On Thu, Oct 10, 2024 at 8:49 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Mikael Sand <msand@seaber.io> writes: > > RUN clang++ -fno-common -static -o main main.cpp \ > > -L/usr/local/lib -lpq -lpgcommon -lpgport \ > > -lldap -lsasl2 -lssl -lcrypto -llber \ > > -lgssapi_krb5 \ > > -lkrb5 -lk5crypto -lcom_err -lkrb5support \ > > -lgdbm > > The short answer here is that your link recipe is wrong, and has been > wrong right along, though you accidentally got away with it before. > The modules within libpq expect to be linked with libpgcommon_shlib > and libpgport_shlib, not libpgcommon/libpgport. > > Having external code that needs to know explicitly about every one > of a library's dependencies is one of many reasons why we discourage > static linking. May I ask what problem you are trying to solve with static linking in the first place? -- Best regards, Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical.
Wonderful, the build works when using -lpgcommon_shlib -lpgport_shlib
So this appears to imply that the output of `pkg-config -libs -static libpq`is incorrect?
/app # pkg-config -libs -static libpq
-L/usr/local/lib -lpq -L/usr/lib/llvm15/lib -L/usr/local/lib -lpgcommon -lpgport -lgssapi_krb5 -lm -lldap -lssl -lcrypto -ldl -pthread
On Thu, Oct 10, 2024 at 7:54 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Mikael,
On Thu, Oct 10, 2024 at 8:49 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Mikael Sand <msand@seaber.io> writes:
> > RUN clang++ -fno-common -static -o main main.cpp \
> > -L/usr/local/lib -lpq -lpgcommon -lpgport \
> > -lldap -lsasl2 -lssl -lcrypto -llber \
> > -lgssapi_krb5 \
> > -lkrb5 -lk5crypto -lcom_err -lkrb5support \
> > -lgdbm
>
> The short answer here is that your link recipe is wrong, and has been
> wrong right along, though you accidentally got away with it before.
> The modules within libpq expect to be linked with libpgcommon_shlib
> and libpgport_shlib, not libpgcommon/libpgport.
>
> Having external code that needs to know explicitly about every one
> of a library's dependencies is one of many reasons why we discourage
> static linking.
May I ask what problem you are trying to solve with static linking in
the first place?
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Just for reference in case anyone else who utilizes static linking for any reason hits upon this issue, here is a working Dockerfile for libpq / postgresql 17
FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
libedit-dev \
openssl-dev \
zstd-static \
zlib-static \
lz4-static \
e2fsprogs \
keyutils \
zstd-dev \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
ARG KRB5=1.21.3
ARG KRB5MAJMIN=1.21
RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz | tar xzf -
RUN cd krb5-$KRB5/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
ARG SASL=2.1.28
RUN curl -L https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz | tar xzf -
RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
RUN clang++ -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Tom Lane
Date:
Mikael Sand <msand@seaber.io> writes: > Wonderful, the build works when using -lpgcommon_shlib -lpgport_shlib > So this appears to imply that the output of `pkg-config -libs -static > libpq`is incorrect? >> /app # pkg-config -libs -static libpq >> -L/usr/local/lib -lpq -L/usr/lib/llvm15/lib -L/usr/local/lib -lpgcommon >> -lpgport -lgssapi_krb5 -lm -lldap -lssl -lcrypto -ldl -pthread Hmm ... does seem that way. Peter apparently deliberately made it so in 55392bc5b, but there's no real justification for removing _shlib in the commit message. Peter? regards, tom lane
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Aleksander, do you have something against static linking or am I reading you wrong?
I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?
At least for executables intended to be run in production inside containers, they usually run in isolation, what other executables could they share libraries with?
Should we leave performance on the table, given the energy and resource requirements it implies?
Certainly, for libraries intended to be distributed in e.g. Linux distributions and be depended on in end-user executables, it makes sense to have shared libraries and simplify mitigating security issues.
It appears to me that there are many cases where static linking and link time optimization would be more appropriate than dynamic.
When the executables never have a chance to share libraries with any other process and are rebuilt whenever a new version is deployed what benefits does dynamic linking provide over static + lto?
I expect I'm missing something crucial, but I have spent some time trying to understand this issue deeper, and still find myself mystified.
Please enlighten me.
Br Mikael
On Thu, Oct 10, 2024 at 8:29 PM Mikael Sand <msand@seaber.io> wrote:
Just for reference in case anyone else who utilizes static linking for any reason hits upon this issue, here is a working Dockerfile for libpq / postgresql 17FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
libedit-dev \
openssl-dev \
zstd-static \
zlib-static \
lz4-static \
e2fsprogs \
keyutils \
zstd-dev \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
ARG KRB5=1.21.3
ARG KRB5MAJMIN=1.21
RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz | tar xzf -
RUN cd krb5-$KRB5/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
ARG SASL=2.1.28
RUN curl -L https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz | tar xzf -
RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
RUN clang++ -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Tom
Would this be an acceptable patch?
Subject: [PATCH] Fix static build
---
Index: src/Makefile.shlib
<+>UTF-8
===================================================================
diff --git a/src/Makefile.shlib b/src/Makefile.shlib
--- a/src/Makefile.shlib (revision fd64ed60b62697984bb69a09a3ae19fbe2905eb6)
+++ b/src/Makefile.shlib (date 1728590127913)
@@ -354,7 +354,7 @@
# those that point inside the build or source tree. Use sort to
# remove duplicates. Also record the -l flags necessary for static
# linking, but not those already covered by Requires.private.
- echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter -L%,$(LDFLAGS) $(SHLIB_LINK)))) $(filter-out $(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter -l%,$(SHLIB_LINK_INTERNAL:%_shlib=%) $(SHLIB_LINK)))' >>$@
+ echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter -L%,$(LDFLAGS) $(SHLIB_LINK)))) $(filter-out $(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter -l%,$(SHLIB_LINK_INTERNAL) $(SHLIB_LINK)))' >>$@
##
---
Index: src/Makefile.shlib
<+>UTF-8
===================================================================
diff --git a/src/Makefile.shlib b/src/Makefile.shlib
--- a/src/Makefile.shlib (revision fd64ed60b62697984bb69a09a3ae19fbe2905eb6)
+++ b/src/Makefile.shlib (date 1728590127913)
@@ -354,7 +354,7 @@
# those that point inside the build or source tree. Use sort to
# remove duplicates. Also record the -l flags necessary for static
# linking, but not those already covered by Requires.private.
- echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter -L%,$(LDFLAGS) $(SHLIB_LINK)))) $(filter-out $(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter -l%,$(SHLIB_LINK_INTERNAL:%_shlib=%) $(SHLIB_LINK)))' >>$@
+ echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter -L%,$(LDFLAGS) $(SHLIB_LINK)))) $(filter-out $(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter -l%,$(SHLIB_LINK_INTERNAL) $(SHLIB_LINK)))' >>$@
##
Br Mikael
On Thu, Oct 10, 2024 at 9:08 PM Mikael Sand <msand@seaber.io> wrote:
Aleksander, do you have something against static linking or am I reading you wrong?I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?At least for executables intended to be run in production inside containers, they usually run in isolation, what other executables could they share libraries with?Should we leave performance on the table, given the energy and resource requirements it implies?Certainly, for libraries intended to be distributed in e.g. Linux distributions and be depended on in end-user executables, it makes sense to have shared libraries and simplify mitigating security issues.It appears to me that there are many cases where static linking and link time optimization would be more appropriate than dynamic.When the executables never have a chance to share libraries with any other process and are rebuilt whenever a new version is deployed what benefits does dynamic linking provide over static + lto?I expect I'm missing something crucial, but I have spent some time trying to understand this issue deeper, and still find myself mystified.Please enlighten me.Br MikaelOn Thu, Oct 10, 2024 at 8:29 PM Mikael Sand <msand@seaber.io> wrote:Just for reference in case anyone else who utilizes static linking for any reason hits upon this issue, here is a working Dockerfile for libpq / postgresql 17FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
libedit-dev \
openssl-dev \
zstd-static \
zlib-static \
lz4-static \
e2fsprogs \
keyutils \
zstd-dev \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <<EOF ./main.cpp
#include<libpq-fe.h>
int main(){return PQconnectdb("")==NULL;}
EOF
ARG KRB5=1.21.3
ARG KRB5MAJMIN=1.21
RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz | tar xzf -
RUN cd krb5-$KRB5/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
ARG SASL=2.1.28
RUN curl -L https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz | tar xzf -
RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
RUN clang++ -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Aleksander Alekseev
Date:
Hi Mikael, > We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-criticalqueries, and it simplifies our security audits to have a static binary running inside chainguard/staticas the data we handle is sensitive/business critical. No comments about the security aspect. I'm not a security expert so I leave this topic for somebody for whom this is an area of expertise. I don't buy the optimization part. The performance gain you get is next to nothing compared to the time spent transferring SQL queries over the network / UDP sockets, parsing and planning them, accessing the disk, waiting for the locks, etc. Unless you actually measured it in order to show the opposite which I seriously doubt you did. A better time investment would be finding actual bottlenecks in the given system in order to be able to eliminate them. > Aleksander, do you have something against static linking or am I reading you wrong? I've seen this sentiment in severalplaces but never understood why anyone would hold this position. Could you elaborate? The question is whether it's practical for the PostgreSQL community to put effort into supporting / testing static linking of libpq. Personally I'm not convinced that demanding from every open-source project in existence to support static linking is a sustainable idea in the long run, given the fact that there are technologies like Docker and programming languages like Go (with pgx client). -- Best regards, Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Hi Aleksander
Ok. So no actual benefit from using dynamic?
Well, it seems postgresql and all dependencies already support it, no?
Doesn't go do static linking by default / prefer it? Unless you use some part that uses CGO, in which case many go developers appear to disable CGO anyway and use the plain go implementation instead.
Similarly, the rust community seems to have a preference / strong support for static, some with alpine, musl, etc.
We're still developing this part of our system, and do intend to measure the difference, for this we need to be able to build the static version as well. I can post results once we have them. Have you measured the difference?
Br Mikael
On Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Hi Mikael,
> We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical.
No comments about the security aspect. I'm not a security expert so I
leave this topic for somebody for whom this is an area of expertise.
I don't buy the optimization part. The performance gain you get is
next to nothing compared to the time spent transferring SQL queries
over the network / UDP sockets, parsing and planning them, accessing
the disk, waiting for the locks, etc. Unless you actually measured it
in order to show the opposite which I seriously doubt you did. A
better time investment would be finding actual bottlenecks in the
given system in order to be able to eliminate them.
> Aleksander, do you have something against static linking or am I reading you wrong? I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?
The question is whether it's practical for the PostgreSQL community to
put effort into supporting / testing static linking of libpq.
Personally I'm not convinced that demanding from every open-source
project in existence to support static linking is a sustainable idea
in the long run, given the fact that there are technologies like
Docker and programming languages like Go (with pgx client).
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Btw Aleksander
Br Mikael
On Thu, Oct 10, 2024 at 10:22 PM Mikael Sand <msand@seaber.io> wrote:
Hi AleksanderOk. So no actual benefit from using dynamic?Well, it seems postgresql and all dependencies already support it, no?Doesn't go do static linking by default / prefer it? Unless you use some part that uses CGO, in which case many go developers appear to disable CGO anyway and use the plain go implementation instead.Similarly, the rust community seems to have a preference / strong support for static, some with alpine, musl, etc.We're still developing this part of our system, and do intend to measure the difference, for this we need to be able to build the static version as well. I can post results once we have them. Have you measured the difference?Br MikaelOn Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev <aleksander@timescale.com> wrote:Hi Mikael,
> We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical.
No comments about the security aspect. I'm not a security expert so I
leave this topic for somebody for whom this is an area of expertise.
I don't buy the optimization part. The performance gain you get is
next to nothing compared to the time spent transferring SQL queries
over the network / UDP sockets, parsing and planning them, accessing
the disk, waiting for the locks, etc. Unless you actually measured it
in order to show the opposite which I seriously doubt you did. A
better time investment would be finding actual bottlenecks in the
given system in order to be able to eliminate them.
> Aleksander, do you have something against static linking or am I reading you wrong? I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?
The question is whether it's practical for the PostgreSQL community to
put effort into supporting / testing static linking of libpq.
Personally I'm not convinced that demanding from every open-source
project in existence to support static linking is a sustainable idea
in the long run, given the fact that there are technologies like
Docker and programming languages like Go (with pgx client).
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
One question,
I wonder if it would be possible to link pgcommon_shlib and pgport_shlib statically into pq and do LTO already at that point, such that consumers of libpq could merely link against libpq.a and that would be enough?
I haven't maintained any c or c++ library intended for public distribution and haven't needed this kind of functionality, but perhaps someone on the mailing list would already have the necessary expertise?
Br Mikael
On Thu, Oct 10, 2024 at 10:33 PM Mikael Sand <msand@seaber.io> wrote:
Btw AleksanderTo give some context, this is a component intended to solve our last scaling issue / most demanding query, which happens at page load and thus need to minimize latency over all else. We've used libpq in pipelined binary mode, serialized the results it in a custom binary format, compressed that using zstd, sent to the frontend, uncompressed by the browser builtin support, and parsed from binary to javascript objects using custom parser. This has improved performance more than 50x over the existing graphql / postgraphile solution so far. It seems likely that doing this work in a read replica would further improve performance significantly, so as not to send the large amount of data from the database to the backend, but this will be a later project. I can have measurements regarding static + lto vs dynamic before that.Br MikaelOn Thu, Oct 10, 2024 at 10:22 PM Mikael Sand <msand@seaber.io> wrote:Hi AleksanderOk. So no actual benefit from using dynamic?Well, it seems postgresql and all dependencies already support it, no?Doesn't go do static linking by default / prefer it? Unless you use some part that uses CGO, in which case many go developers appear to disable CGO anyway and use the plain go implementation instead.Similarly, the rust community seems to have a preference / strong support for static, some with alpine, musl, etc.We're still developing this part of our system, and do intend to measure the difference, for this we need to be able to build the static version as well. I can post results once we have them. Have you measured the difference?Br MikaelOn Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev <aleksander@timescale.com> wrote:Hi Mikael,
> We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical.
No comments about the security aspect. I'm not a security expert so I
leave this topic for somebody for whom this is an area of expertise.
I don't buy the optimization part. The performance gain you get is
next to nothing compared to the time spent transferring SQL queries
over the network / UDP sockets, parsing and planning them, accessing
the disk, waiting for the locks, etc. Unless you actually measured it
in order to show the opposite which I seriously doubt you did. A
better time investment would be finding actual bottlenecks in the
given system in order to be able to eliminate them.
> Aleksander, do you have something against static linking or am I reading you wrong? I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?
The question is whether it's practical for the PostgreSQL community to
put effort into supporting / testing static linking of libpq.
Personally I'm not convinced that demanding from every open-source
project in existence to support static linking is a sustainable idea
in the long run, given the fact that there are technologies like
Docker and programming languages like Go (with pgx client).
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Furthermore, the queries in this case are 10667 bytes in total, so should fit inside ten-ish MTUs in an AWS VPC with the pg wire protocol overhead. So I'm not too worried about the time spent sending the queries. Still, it would be interesting to measure the impact of storing them in the DB and pre-computing them as far as possible without authorization information. We've planned further profiling and optimization of data access and physical layout. No locks should affect this specific use case.
On Thu, Oct 10, 2024 at 11:07 PM Mikael Sand <msand@seaber.io> wrote:
One question,I wonder if it would be possible to link pgcommon_shlib and pgport_shlib statically into pq and do LTO already at that point, such that consumers of libpq could merely link against libpq.a and that would be enough?I haven't maintained any c or c++ library intended for public distribution and haven't needed this kind of functionality, but perhaps someone on the mailing list would already have the necessary expertise?Br MikaelOn Thu, Oct 10, 2024 at 10:33 PM Mikael Sand <msand@seaber.io> wrote:Btw AleksanderTo give some context, this is a component intended to solve our last scaling issue / most demanding query, which happens at page load and thus need to minimize latency over all else. We've used libpq in pipelined binary mode, serialized the results it in a custom binary format, compressed that using zstd, sent to the frontend, uncompressed by the browser builtin support, and parsed from binary to javascript objects using custom parser. This has improved performance more than 50x over the existing graphql / postgraphile solution so far. It seems likely that doing this work in a read replica would further improve performance significantly, so as not to send the large amount of data from the database to the backend, but this will be a later project. I can have measurements regarding static + lto vs dynamic before that.Br MikaelOn Thu, Oct 10, 2024 at 10:22 PM Mikael Sand <msand@seaber.io> wrote:Hi AleksanderOk. So no actual benefit from using dynamic?Well, it seems postgresql and all dependencies already support it, no?Doesn't go do static linking by default / prefer it? Unless you use some part that uses CGO, in which case many go developers appear to disable CGO anyway and use the plain go implementation instead.Similarly, the rust community seems to have a preference / strong support for static, some with alpine, musl, etc.We're still developing this part of our system, and do intend to measure the difference, for this we need to be able to build the static version as well. I can post results once we have them. Have you measured the difference?Br MikaelOn Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev <aleksander@timescale.com> wrote:Hi Mikael,
> We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical.
No comments about the security aspect. I'm not a security expert so I
leave this topic for somebody for whom this is an area of expertise.
I don't buy the optimization part. The performance gain you get is
next to nothing compared to the time spent transferring SQL queries
over the network / UDP sockets, parsing and planning them, accessing
the disk, waiting for the locks, etc. Unless you actually measured it
in order to show the opposite which I seriously doubt you did. A
better time investment would be finding actual bottlenecks in the
given system in order to be able to eliminate them.
> Aleksander, do you have something against static linking or am I reading you wrong? I've seen this sentiment in several places but never understood why anyone would hold this position. Could you elaborate?
The question is whether it's practical for the PostgreSQL community to
put effort into supporting / testing static linking of libpq.
Personally I'm not convinced that demanding from every open-source
project in existence to support static linking is a sustainable idea
in the long run, given the fact that there are technologies like
Docker and programming languages like Go (with pgx client).
--
Best regards,
Aleksander Alekseev
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Tom Lane
Date:
Mikael Sand <msand@seaber.io> writes: > I wonder if it would be possible to link pgcommon_shlib and pgport_shlib > statically into pq and do LTO already at that point, such that consumers of > libpq could merely link against libpq.a and that would be enough? What would be the point? A typical build will still have a pile of outside dependencies that static-link consumers will have to track manually (openssl and so forth). So I don't see how this'd move the needle much. regards, tom lane
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
From
Mikael Sand
Date:
Well for getting the potential benefits of link-time optimization into more places, and slightly improved developer ergonomics, and perhaps mostly to eliminate potential confusion if one needs pg_common and pg_ports or the shared library version when compiling statically.
Would you happen to know if it is possible?
regards, mikael sand
On Fri, Oct 11, 2024 at 12:12 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Mikael Sand <msand@seaber.io> writes:
> I wonder if it would be possible to link pgcommon_shlib and pgport_shlib
> statically into pq and do LTO already at that point, such that consumers of
> libpq could merely link against libpq.a and that would be enough?
What would be the point? A typical build will still have a pile of
outside dependencies that static-link consumers will have to track
manually (openssl and so forth). So I don't see how this'd move
the needle much.
regards, tom lane