Thread: statically compiling postgres and problem with initdb

On 1/07/2011 3:28 PM, mona attariyan wrote:
> Hi,
> I statically compiled my postgres and I got the following message when I
> ran initdb:

How did you compile Pg statically?

Please show your configure command line, any environment variables you
set, etc.

You've also left out a huge amount of information, like your Pg version,
where you got the sources from, your OS version, your compiler version,
etc. I strongly suggest that you read this:

   http://wiki.postgresql.org/wiki/Guide_to_reporting_problems

before posting a follow-up.

> There was a discussion about this in the archive, but there wasn't any
> fix for it. I have to compile postgres statically for the project I'm
> doing.

Why? Perhaps if you explained that in a bit more detail it'd be possible
to find an alternative.

--
Craig Ringer

On 1/07/2011 5:11 PM, mona attariyan wrote:

 > CFLAGS="-O0 -I/[PATH]/glibc-2.5.1-custom/prefix/include -static

Yeah, it's not as simple as that, because Pg expects to be dynamically
linked and to be able to dlopen() libraries and dlsym() resolve function
pointers from them at runtime.

 > creating conversions ... FATAL: could not load library
 > "/usr/local/pgsql/lib/ascii_and_mic.so":
 > /usr/local/pgsql/lib/ascii_and_mic.so: undefined symbol: pg_ascii2mic

... like that. You'll have to modify Pg to statically link such required
libraries in, and add a new lookup method that can return function
pointers to the internal statically linked copies of the functions when
they're required.

> I'm using Postgres to evaluate a research tool and the tool doesn't work
> with dynamic libraries. That's why I need to compile it statically.

OK, so you're not trying to embed Pg into an application or make it run
from a single executable. You just need "postgres", initdb, etc to each
be a statically linked executable. That's good; some people who ask that
sort of thing are angling for embedding it in an app, and it really
won't work.

For your purposes, you'll have to modify PostgreSQL to support being
built statically. It *expects* to be able to dlopen() libraries at
runtime, and those libraries often expect to be able to resolve symbols
from the postgres executable they're being linked into. This won't work
when it's a static binary. You'll have to figure out which libraries
each Pg executable needs during its lifetime, statically link them in as
well, and modify postgresql to know which libraries are linked in at
startup so it doesn't try to dlopen() them.

You can probably get Pg to use static copies of libraries it normally
dlopen()s by building mappings of library names to structs full of
function pointers that you return instead of the usual dlysm()'d
function pointer from a dlopen()'d library. If you do it that way, the
rest of Pg might not even need to know it's not truly dynamically
loading things.

--
Craig Ringer

Craig Ringer <craig@postnewspapers.com.au> writes:
> On 1/07/2011 5:11 PM, mona attariyan wrote:
>> I'm using Postgres to evaluate a research tool and the tool doesn't work
>> with dynamic libraries. That's why I need to compile it statically.

> For your purposes, you'll have to modify PostgreSQL to support being
> built statically.

It might be easier to rip out the functionality that expects loadable
libraries to work.  I think you could probably get through initdb if you
just disabled creation of encoding-conversion functions and text search
dictionaries (try #ifdef'ing out the relevant sections of initdb.c).

Of course, you'll end up with a pretty crippled version of PG --- no
encoding conversions, no text search, no procedural languages --- but
maybe that's enough for what you want to do.  If it's not, then as Craig
says, you're looking at some pretty major work to bind those pieces into
the executable statically.

            regards, tom lane

On 1/07/2011 11:48 PM, Tom Lane wrote:

> It might be easier to rip out the functionality that expects loadable
> libraries to work.  I think you could probably get through initdb if you
> just disabled creation of encoding-conversion functions and text search
> dictionaries (try #ifdef'ing out the relevant sections of initdb.c).

Good point. If all that's needed is basic functionality...

> Of course, you'll end up with a pretty crippled version of PG --- no
> encoding conversions, no text search, no procedural languages --- but
> maybe that's enough for what you want to do.  If it's not, then as Craig
> says, you're looking at some pretty major work to bind those pieces into
> the executable statically.

I had to do something quite similar for Scribus years ago, and it wasn't
anywhere near as hard as I'd feared. It did have two  really annoying
bits, though. One was having to prefix each shared library's public
symbols with the name of the shared library to avoid conflicts and allow
me to differentiate different implementations of the same public
interfaces. The second was modifying the build system to link each
library to the main executable.

This is from increasingly vague memory, but:

First I created a function pointer list entry struct type that maps
function names to function pointers.

I then modified the loader code so it prefixed the function names it was
expecting with the library name. Instead of "funcname" it'd try to
resolve "libname_funcname".

I went through EVERY SINGLE LIBRARY and prefixed the library name to the
names of every non-static function. It was ugly, but the alternative
would've been a horrid token-pasting macro hack that would've still
required changing each function declaration.

The libraries already had headers. If they hadn't, I would've had to
write a header for each library that declared prototypes for its functions.

I added a new header file to the main build that included all the
library headers and declared a global array of function pointer list
entries. In the associated .c (well, .cxx in Scribus) I defined the
array, populating it with the library-name-prefixed function names and
pointers to each function.

I modified each site where Scribus used dlopen() and dlsym() so it
called them via a wrapper. The wrapper for dynamic linking was just a
trivial header of inline wrappers around dlopen() and dlsym(). The
static linking replacement for dlopen() just returned the input library
name char* as void*, and the dlsym() replacement cast the void* back to
char*, joined the library name and function name, looked the result up
in the function pointer list, and returned the resulting function pointer.

Finally, I had to modify the build process so it produced static
libraries instead of shared libraries for each add-on, and modify the
final application linkage so it linked each library. This was made a lot
easier by the fact that Scribus used CMake instead of autotools; I don't
know what it'd be like to try to do this with autohell, but I suspect
"ugly" would be a start.

--
Craig Ringe