Thread: static or dynamic libpgport
Recently I attempted to build an external package (pg_bulkload) against the latest Fedora packages. Unfortunately this fails, as pgxs adds "-lpgport" to any link line for an executable, and the corresponding libpgport.a isn't there. And in fact, pg_bulkload does use some of the functionality there (e.g. pg_strncasecmp), so just stripping "-lpgport" out doesn't work either. This happened because Fedora packaging guidelines <http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries> are strongly against shipping static libraries, and so all the PostgreSQL static libraries are excluded from the distribution (and I believe there are similar restrictions for RHEL). Of these libraries, I believe the only one that is *only* built as a static library is libpgport. Is there any good reason why we shouldn't build and install a dynamic libpgport.so? (Of course, you could say "use the community RPMs", but that would be a bit of a cop out. Some organizations have a perfectly reasonable policy or requiring use of vendor packages wherever possible, since vendors are naturally only going to support packages they provide. So either we should be arguing to the Fedora/RedHat people that they should ship the static library, or we should be providing them with a dynamic one, ISTM.) cheers andrew
On 9 December 2011 16:13, Andrew Dunstan <andrew@dunslane.net> wrote: > Is there any good reason why we shouldn't build and install a dynamic > libpgport.so? +1 in favour of building and installing a dynamic libpgport.so. I generally agree with your analysis. I've seen this issue crop up a good few times now. I'm a Fedora user myself, but about 2 years ago I got into a "he said she said" situation with an OpenSUSE package maintainer over this, when I had to build Slony on that platform. I'm a bit hazy on the details now, but iirc he thought that it wasn't necessary to ship libpgport.a in particular (though I don't think that they have a beef with static libraries generally) - maybe they took a cue from Redhat there? -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services
On 11-12-09 11:13 AM, Andrew Dunstan wrote: > Recently I attempted to build an external package (pg_bulkload) > against the latest Fedora packages. Unfortunately this fails, as pgxs > adds "-lpgport" to any link line for an executable, and the > corresponding libpgport.a isn't there. And in fact, pg_bulkload does > use some of the functionality there (e.g. pg_strncasecmp), so just > stripping "-lpgport" out doesn't work either. > > This happened because Fedora packaging guidelines > <http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries> > are strongly against shipping static libraries, and so all the > PostgreSQL static libraries are excluded from the distribution (and I > believe there are similar restrictions for RHEL). Of these libraries, > I believe the only one that is *only* built as a static library is > libpgport. > > Is there any good reason why we shouldn't build and install a dynamic > libpgport.so? +1 We've struggled with slony and pgport because so many users have had problems with pgport not being included in some distributions. It has some useful functions, I think recent versions of slony use it on win32 but don't elsewhere. Wee have had at least one patch floating around that makes conditionally includes certain small behaviours in slony based on if pgport is available or not based on a configure check. What package would a shared static pgport be installed with? Slony requires a server + headers to build but slon and slonik only have a runtime dependency on libpq (I don't know if anyone installs slon/slonik on a machine without a postgresql server but you could) Steve > cheers > > andrew >
On 12/09/2011 01:01 PM, Steve Singer wrote: > On 11-12-09 11:13 AM, Andrew Dunstan wrote: >> Recently I attempted to build an external package (pg_bulkload) >> against the latest Fedora packages. Unfortunately this fails, as pgxs >> adds "-lpgport" to any link line for an executable, and the >> corresponding libpgport.a isn't there. And in fact, pg_bulkload does >> use some of the functionality there (e.g. pg_strncasecmp), so just >> stripping "-lpgport" out doesn't work either. >> >> This happened because Fedora packaging guidelines >> <http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries> >> are strongly against shipping static libraries, and so all the >> PostgreSQL static libraries are excluded from the distribution (and I >> believe there are similar restrictions for RHEL). Of these libraries, >> I believe the only one that is *only* built as a static library is >> libpgport. >> >> Is there any good reason why we shouldn't build and install a dynamic >> libpgport.so? > > +1 > > We've struggled with slony and pgport because so many users have had > problems with pgport not being included in some distributions. It has > some useful functions, I think recent versions of slony use it on > win32 but don't elsewhere. Wee have had at least one patch floating > around that makes conditionally includes certain small behaviours in > slony based on if pgport is available or not based on a configure check. > > What package would a shared static pgport be installed with? Slony > requires a server + headers to build but slon and slonik only have a > runtime dependency on libpq (I don't know if anyone installs > slon/slonik on a machine without a postgresql server but you could) > > In the Fedora world, a static lib would go in postgresql-devel, but a dynamic lib would go in postgresql-libs, which is also where libpq is shipped. cheers andrew
Andrew Dunstan wrote: > >> Is there any good reason why we shouldn't build and install a dynamic > >> libpgport.so? > > > > +1 > > > > We've struggled with slony and pgport because so many users have had > > problems with pgport not being included in some distributions. It has > > some useful functions, I think recent versions of slony use it on > > win32 but don't elsewhere. Wee have had at least one patch floating > > around that makes conditionally includes certain small behaviours in > > slony based on if pgport is available or not based on a configure check. > > > > What package would a shared static pgport be installed with? Slony > > requires a server + headers to build but slon and slonik only have a > > runtime dependency on libpq (I don't know if anyone installs > > slon/slonik on a machine without a postgresql server but you could) > > > > > > In the Fedora world, a static lib would go in postgresql-devel, but a > dynamic lib would go in postgresql-libs, which is also where libpq is > shipped. I am not against shipping a dynamic libpgport, but I will just point out that this was never intended or anticipated. Are there any symbols in there that might conflict with other software? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 12/09/2011 06:27 PM, Bruce Momjian wrote: >> In the Fedora world, a static lib would go in postgresql-devel, but a >> dynamic lib would go in postgresql-libs, which is also where libpq is >> shipped. > I am not against shipping a dynamic libpgport, but I will just point out > that this was never intended or anticipated. Are there any symbols in > there that might conflict with other software? Possibly. Below is a list of symbols from a recent build. The other thing is we'd need to turn on flags that make the object suitable for a dynamic library (e.g. -fpic). I'm not sure if that has any significant impact - probably not or else people would avoid using them far more. I think we should deal with this. Just Peter's and Steve's responses upthread indicate a demand, and I came into this because a customer encountered enormous difficulty in doing an out of tree build of a well known piece of support software, so this is clearly a pain point. If we're not going to do it, we should probably think about adjusting people's expectations. [andrew@diego port]$ nm libpgport.a | grep ' T ' 00000000 T strlcat 00000000 T strlcpy 00000000 T getpeereid 00000000T pg_get_encoding_from_locale 00000000 T pgfnames 000001a0 T pgfnames_cleanup 000001e0 T rmtree 00000240T find_my_exec 000005f0 T find_other_exec 00000500 T pclose_check 00000760 T set_pglocale_pgservice 00000000T inet_net_ntop 00000030 T pg_set_block 00000000 T pg_set_noblock 00000280 T canonicalize_path 00000110 Tfirst_dir_separator 00000140 T first_path_var_separator 00000800 T get_doc_path 00000720 T get_etc_path 00000860T get_home_path 00000820 T get_html_path 00000740 T get_include_path 00000780 T get_includeserver_path 000007a0T get_lib_path 000007e0 T get_locale_path 00000840 T get_man_path 000008e0 T get_parent_directory 00000760T get_pkginclude_path 000007c0 T get_pkglib_path 00000690 T get_progname 00000700 T get_share_path 000001b0T join_path_components 00000170 T last_dir_separator 000001a0 T make_native_path 00000560 T path_contains_parent_reference 00000630 T path_is_prefix_of_path 00000610 T path_is_relative_and_below_cwd 00000000T pg_check_dir 00000000 T pg_mkdir_p 00000000 T pg_usleep 00000330 T pg_ascii_tolower 00000310 T pg_ascii_toupper 00000000 T pg_strcasecmp 00000100 T pg_strncasecmp 00000290 T pg_tolower 00000210 T pg_toupper 000000d0 T pg_qsort 000000f0 T qsort_arg 00000000 T simple_prompt 00000010 T pqGetpwuid 00000000 T pqStrerror cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> On 12/09/2011 06:27 PM, Bruce Momjian wrote:
>> I am not against shipping a dynamic libpgport, but I will just point out
>> that this was never intended or anticipated. Are there any symbols in
>> there that might conflict with other software?
> Possibly. Below is a list of symbols from a recent build.
This doesn't seem like much of an issue to me, since anything wanting to
link against libpgport would be designed to work with whatever it
provides, no?
> The other
> thing is we'd need to turn on flags that make the object suitable for a
> dynamic library (e.g. -fpic).
Right now, libpq laboriously rebuilds all the .o files it needs from
src/port/ so as to get them with -fpic. It would be nice if we could
clean that up while we're doing this. It might be all right to always
build the client-side version of libpgport with -fpic, though I'd be sad
if that leaked into the server-side build.
regards, tom lane
On 12/10/2011 08:26 PM, Tom Lane wrote: > >> The other >> thing is we'd need to turn on flags that make the object suitable for a >> dynamic library (e.g. -fpic). > Right now, libpq laboriously rebuilds all the .o files it needs from > src/port/ so as to get them with -fpic. It would be nice if we could > clean that up while we're doing this. It might be all right to always > build the client-side version of libpgport with -fpic, though I'd be sad > if that leaked into the server-side build. > Here's a small diff that seems to build things the right way. No leakage of -fpic into the server side code. Still a deal of work to do, but it's a start. Would we want to link our own non-backend executables against the shared lib? That would almost certainly break the buildfarm for Windows builds, as it only currently copies the libpq DLL into the bin directory. That's no reason on its own not to do it, of course, and there are only a couple of owners other than me anyway, so it would be easy to fix. How do you want to proceed for libpq (and the ecpg library cases that do the same thing)? Just link in the object files directly? cheers andrew *** Makefile 2011-12-03 17:21:59.944509111 -0500 --- GNUmakefile 2011-12-12 12:39:43.176260505 -0500 *************** *** 37,47 **** # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! all: libpgport.a libpgport_srv.a ! # libpgport is needed by some contrib ! install: all installdirs ! $(INSTALL_STLIB) libpgport.a '$(DESTDIR)$(libdir)/libpgport.a' installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' --- 37,52 ---- # foo_srv.o and foo.o are both built from foo.c, but only foo.o has -DFRONTEND OBJS_SRV = $(OBJS:%.o=%_srv.o) ! NAME = pgport ! SO_MAJOR_VERSION= 1 ! SO_MINOR_VERSION= 1 ! include $(top_srcdir)/src/Makefile.shlib ! ! all: all-lib libpgport_srv.a ! ! # libpgport is needed by any exe built with pgxs ! install: all installdirs install-lib installdirs: $(MKDIR_P) '$(DESTDIR)$(libdir)' *************** *** 49,57 **** uninstall: rm -f '$(DESTDIR)$(libdir)/libpgport.a' - libpgport.a: $(OBJS) - $(AR) $(AROPT) $@ $^ - # thread.o needs PTHREAD_CFLAGS (but thread_srv.o does not) thread.o: thread.c $(CC) $(CFLAGS) $(CPPFLAGS) $(PTHREAD_CFLAGS)-c $< --- 54,59 ---- *************** *** 64,70 **** $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(CFLAGS) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $< -o $@ $(OBJS_SRV): | submake-errcodes --- 66,72 ---- $(AR) $(AROPT) $@ $^ %_srv.o: %.c ! $(CC) $(subst $(CFLAGS_SL),,$(CFLAGS)) $(subst -DFRONTEND,, $(CPPFLAGS)) -c $< -o $@ $(OBJS_SRV): | submake-errcodes *************** *** 97,100 **** echo "#define MANDIR \"$(mandir)\"" >>$@ clean distclean maintainer-clean: ! rm -f libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h --- 99,102 ---- echo "#define MANDIR \"$(mandir)\"" >>$@ clean distclean maintainer-clean: ! rm -f libpgport.so* libpgport.a libpgport_srv.a $(OBJS) $(OBJS_SRV) pg_config_paths.h
On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: > > The other > > thing is we'd need to turn on flags that make the object suitable for a > > dynamic library (e.g. -fpic). > > Right now, libpq laboriously rebuilds all the .o files it needs from > src/port/ so as to get them with -fpic. It would be nice if we could > clean that up while we're doing this. It might be all right to always > build the client-side version of libpgport with -fpic, though I'd be sad > if that leaked into the server-side build. So would we continue to build the client binaries (psql, pg_dump, etc.) against the static libpgport.a, thus keeping it "invisible" there, or would we dynamically link them, thus creating a new dependency.
On fre, 2011-12-09 at 11:13 -0500, Andrew Dunstan wrote: > Is there any good reason why we shouldn't build and install a dynamic > libpgport.so? Just note, if you do this, you need to carefully manage API, ABI, soname, symbol list, and all that. Every time you tweak configure's decision about when to include a replacement function, you need to change the library version. Every time you remove a function, you need to change the soname. Every backpatched portability fix has the potential to escalate to a full shared library versioning dance. Downstream packagers will be delighted, especially if this requires changing the package name every three minor releases. To see what this can lead to in the extreme, check the dependencies that bind has on its internal libraries: bind9 depends: libbind9-60, libdns69, libisc62, libisccc60, libisccfg62, liblwres60
Peter Eisentraut <peter_e@gmx.net> writes:
> On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote:
>> Right now, libpq laboriously rebuilds all the .o files it needs from
>> src/port/ so as to get them with -fpic. It would be nice if we could
>> clean that up while we're doing this. It might be all right to always
>> build the client-side version of libpgport with -fpic, though I'd be sad
>> if that leaked into the server-side build.
> So would we continue to build the client binaries (psql, pg_dump, etc.)
> against the static libpgport.a, thus keeping it "invisible" there, or
> would we dynamically link them, thus creating a new dependency.
I think that if possible we should avoid creating a new dependency for
either the client binaries or libpq.so itself; what I suggest above
is only a simplification of the build process for libpq. If we create
a new dependency we risk packagers breaking things by forgetting to
include it.
The Fedora/RHEL rule against static libraries is meant to prevent
situations where changes in a library would require rebuilding other
packages to get the fixes in place. If we had to make a quick security
fix in libpq, for example, it would suck if dozens of other packages had
to be rebuilt to propagate the change everywhere. However, I don't think
that concern applies to programs that are in the same source package as
the library --- they'd get rebuilt anyway. So I see nothing wrong with
continuing to statically link these .o files into files belonging to the
postgresql package. It's just that I can't export them in a .a file for
*other* source packages to use.
(Whether a security issue in libpgport is really likely to happen is not
a question that this policy concerns itself with ...)
regards, tom lane
On 12/12/2011 02:59 PM, Tom Lane wrote: > Peter Eisentraut<peter_e@gmx.net> writes: >> On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: >>> Right now, libpq laboriously rebuilds all the .o files it needs from >>> src/port/ so as to get them with -fpic. It would be nice if we could >>> clean that up while we're doing this. It might be all right to always >>> build the client-side version of libpgport with -fpic, though I'd be sad >>> if that leaked into the server-side build. >> So would we continue to build the client binaries (psql, pg_dump, etc.) >> against the static libpgport.a, thus keeping it "invisible" there, or >> would we dynamically link them, thus creating a new dependency. > I think that if possible we should avoid creating a new dependency for > either the client binaries or libpq.so itself; what I suggest above > is only a simplification of the build process for libpq. If we create > a new dependency we risk packagers breaking things by forgetting to > include it. > > OK, I'll work on this basis. The downside is that we'll be building it but not using it, but I can see the advantages. cheers andrew
On 12/12/2011 02:59 PM, Tom Lane wrote: > Peter Eisentraut<peter_e@gmx.net> writes: >> On lör, 2011-12-10 at 20:26 -0500, Tom Lane wrote: >>> Right now, libpq laboriously rebuilds all the .o files it needs from >>> src/port/ so as to get them with -fpic. It would be nice if we could >>> clean that up while we're doing this. It might be all right to always >>> build the client-side version of libpgport with -fpic, though I'd be sad >>> if that leaked into the server-side build. >> So would we continue to build the client binaries (psql, pg_dump, etc.) >> against the static libpgport.a, thus keeping it "invisible" there, or >> would we dynamically link them, thus creating a new dependency. > I think that if possible we should avoid creating a new dependency for > either the client binaries or libpq.so itself; what I suggest above > is only a simplification of the build process for libpq. If we create > a new dependency we risk packagers breaking things by forgetting to > include it. > > The Fedora/RHEL rule against static libraries is meant to prevent > situations where changes in a library would require rebuilding other > packages to get the fixes in place. If we had to make a quick security > fix in libpq, for example, it would suck if dozens of other packages had > to be rebuilt to propagate the change everywhere. However, I don't think > that concern applies to programs that are in the same source package as > the library --- they'd get rebuilt anyway. So I see nothing wrong with > continuing to statically link these .o files into files belonging to the > postgresql package. It's just that I can't export them in a .a file for > *other* source packages to use. > > (Whether a security issue in libpgport is really likely to happen is not > a question that this policy concerns itself with ...) > > OK, my possibly naïve approach is shown in the attached. Essentially it builds libpgport-shared.so and then installs it as libpgport.so. That ensures that the library is not used in building any postgres binaries or libraries. Places such as libpq that formerly symlinked and recompiled the sources in a way that is suitable for a shared library now just link in the already built object files. Is there a better way to do this? cheers andrew