Supporting plpython 2+3 builds better - Mailing list pgsql-hackers

I wrote:
> After reading the recent thread about python 2 vs python 3 support,
> I thought I'd amuse myself by trying to get plpython3 supported in
> the Fedora packages.  That turned out to be unreasonably painful
> (which is something we oughta fix eventually), but it worked,

To give you an idea of what "unreasonably painful" means, attached is
the specfile diff needed to make this happen.  I will not comment on
the fragility of this beyond observing that the "touch -r" commands
are *necessary*, else you get incorrect results.

I think we really need to do something to make this type of build less
painful, because as far as I can see most distros are going to be
wanting to support both python 2 and 3 for awhile to come.

A sketch of a solution might go like this:

* Extend configure to have two switches, say "--with-python" and
"--with-python3", which you can select either or both of.  If you
want to pick executables that are not named "python" and "python3",
perhaps write it like "--with-python3=/usr/bin/python3.2mu".  (I'd
be inclined to remove the dependency on a PYTHON envvar altogether.)

* Make configure output two independent sets of variables into
Makefile.global, viz

    python2_includespec    = ...
    python2_libdir        = ...
    python2_libspec        = ...
    python2_additional_libs    = ...
    python2_configdir    = ...
    python2_majorversion    = ...
    python2_version        = ...

    python3_includespec    = ...
    python3_libdir        = ...
    python3_libspec        = ...
    python3_additional_libs    = ...
    python3_configdir    = ...
    python3_majorversion    = ...
    python3_version        = ...

* Make plpython's Makefile build, test, install plpython2 and/or
plpython3 depending on what's set in Makefile.global.

I'm not sure yet whether it's possible to do this without duplicating a
lot of logic in config/python.m4 and plpython's Makefile.

Another problem is that Makefile.shlib isn't designed to build more than
one shared library per directory, which means that we might end up
having to copy all the source files into a new plpython3 subdirectory
anyway.  We're already doing some of that with the regression test
files, though.

Thoughts?

            regards, tom lane

diff --git a/postgresql.spec b/postgresql.spec
index 524d81c..62ce352 100644
*** a/postgresql.spec
--- b/postgresql.spec
***************
*** 35,40 ****
--- 35,41 ----
  %{!?test:%global test 1}
  %{!?upgrade:%global upgrade 1}
  %{!?plpython:%global plpython 1}
+ %{!?plpython3:%global plpython3 1}
  %{!?pltcl:%global pltcl 1}
  %{!?plperl:%global plperl 1}
  %{!?ssl:%global ssl 1}
*************** BuildRequires: systemd-units
*** 107,112 ****
--- 108,117 ----
  BuildRequires: python-devel
  %endif

+ %if %plpython3
+ BuildRequires: python3-devel
+ %endif
+
  %if %pltcl
  BuildRequires: tcl-devel
  %endif
*************** Install this if you want to write databa
*** 265,278 ****

  %if %plpython
  %package plpython
! Summary: The Python procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}

  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python.
  %endif

  %if %pltcl
--- 270,295 ----

  %if %plpython
  %package plpython
! Summary: The Python2 procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}

  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 2.
! %endif
!
! %if %plpython3
! %package plpython3
! Summary: The Python3 procedural language for PostgreSQL
! Group: Applications/Databases
! Requires: %{name}-server%{?_isa} = %{version}-%{release}
!
! %description plpython3
! The postgresql-plpython3 package contains the PL/Python3 procedural language,
! which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 3.
  %endif

  %if %pltcl
*************** CFLAGS="$CFLAGS -DLINUX_OOM_SCORE_ADJ=0"
*** 346,351 ****
--- 363,447 ----
  # CFLAGS=`echo $CFLAGS| sed -e "s|-O2|-O1|g" `
  # %%endif

+ # plpython requires separate configure/build runs to build against python 2
+ # versus python 3.  Our strategy is to do the python 3 run first, then make
+ # distclean and do it again for the "normal" build.  Note that the installed
+ # Makefile.global will reflect the python 2 build, which seems appropriate
+ # since that's still considered the default plpython version.
+ %if %plpython3
+
+ export PYTHON=/usr/bin/python3
+
+ # These configure options must match main build
+ %configure --disable-rpath \
+ %if %beta
+     --enable-debug \
+     --enable-cassert \
+ %endif
+ %if %plperl
+     --with-perl \
+ %endif
+ %if %pltcl
+     --with-tcl \
+     --with-tclconfig=%{_libdir} \
+ %endif
+ %if %plpython3
+     --with-python \
+ %endif
+ %if %ldap
+     --with-ldap \
+ %endif
+ %if %ssl
+     --with-openssl \
+ %endif
+ %if %pam
+     --with-pam \
+ %endif
+ %if %kerberos
+     --with-krb5 \
+     --with-gssapi \
+ %endif
+ %if %uuid
+     --with-ossp-uuid \
+ %endif
+ %if %xml
+     --with-libxml \
+     --with-libxslt \
+ %endif
+ %if %nls
+     --enable-nls \
+ %endif
+ %if %sdt
+     --enable-dtrace \
+ %endif
+ %if %selinux
+     --with-selinux \
+ %endif
+     --with-system-tzdata=/usr/share/zoneinfo \
+     --datadir=/usr/share/pgsql
+
+ # Fortunately we don't need to build much except plpython itself
+ cd src/backend
+ make submake-errcodes
+ cd ../..
+ cd src/pl/plpython
+ make %{?_smp_mflags} all
+ cd ..
+ # save built form in a directory that "make distclean" won't touch
+ cp -a plpython plpython3
+ cd ../..
+
+ # must also save this version of Makefile.global for later
+ cp src/Makefile.global src/Makefile.global.python3
+
+ make distclean
+
+ %endif
+
+ unset PYTHON
+
+ # Normal (not python3) build begins here
+
  %configure --disable-rpath \
  %if %beta
      --enable-debug \
*************** rm -f src/tutorial/GNUmakefile
*** 409,414 ****
--- 505,528 ----
      pushd src/pl
      make MAX_CONNECTIONS=5 check
      popd
+ %if %plpython3
+     # must install Makefile.global that selects python3
+     mv src/Makefile.global src/Makefile.global.save
+     cp src/Makefile.global.python3 src/Makefile.global
+     touch -r src/Makefile.global.save src/Makefile.global
+     # because "make check" does "make install" on the whole tree,
+     # we must temporarily install plpython3 as src/pl/plpython,
+     # since that is the subdirectory src/pl/Makefile knows about
+     mv src/pl/plpython src/pl/plpython2
+     mv src/pl/plpython3 src/pl/plpython
+     pushd src/pl/plpython
+     make MAX_CONNECTIONS=5 check
+     popd
+     # and clean up our mess
+     mv src/pl/plpython src/pl/plpython3
+     mv src/pl/plpython2 src/pl/plpython
+     mv -f src/Makefile.global.save src/Makefile.global
+ %endif
      pushd contrib
      make MAX_CONNECTIONS=5 check
      popd
*************** rm -f src/tutorial/GNUmakefile
*** 442,447 ****
--- 556,571 ----

  make DESTDIR=$RPM_BUILD_ROOT install-world

+ %if %plpython3
+     mv src/Makefile.global src/Makefile.global.save
+     cp src/Makefile.global.python3 src/Makefile.global
+     touch -r src/Makefile.global.save src/Makefile.global
+     pushd src/pl/plpython3
+     make DESTDIR=$RPM_BUILD_ROOT install
+     popd
+     mv -f src/Makefile.global.save src/Makefile.global
+ %endif
+
  # multilib header hack; note pg_config.h is installed in two places!
  # we only apply this to known Red Hat multilib arches, per bug #177564
  case `uname -i` in
*************** cp /dev/null devel.lst
*** 588,593 ****
--- 713,719 ----
  cp /dev/null plperl.lst
  cp /dev/null pltcl.lst
  cp /dev/null plpython.lst
+ cp /dev/null plpython3.lst

  %if %nls
  %find_lang ecpg-%{majorversion}
*************** cat plpgsql-%{majorversion}.lang >>serve
*** 622,627 ****
--- 748,758 ----
  %find_lang plpython-%{majorversion}
  cat plpython-%{majorversion}.lang >>plpython.lst
  %endif
+ %if %plpython3
+ # plpython3 shares message files with plpython
+ %find_lang plpython-%{majorversion}
+ cat plpython-%{majorversion}.lang >>plpython3.lst
+ %endif
  %if %pltcl
  %find_lang pltcl-%{majorversion}
  cat pltcl-%{majorversion}.lang >>pltcl.lst
*************** fi
*** 907,916 ****

  %if %plpython
  %files plpython -f plpython.lst
! %{_datadir}/pgsql/extension/plpython*
  %{_libdir}/pgsql/plpython2.so
  %endif

  %if %test
  %files test
  %defattr(-,postgres,postgres)
--- 1051,1067 ----

  %if %plpython
  %files plpython -f plpython.lst
! %{_datadir}/pgsql/extension/plpythonu*
! %{_datadir}/pgsql/extension/plpython2*
  %{_libdir}/pgsql/plpython2.so
  %endif

+ %if %plpython3
+ %files plpython3 -f plpython3.lst
+ %{_datadir}/pgsql/extension/plpython3*
+ %{_libdir}/pgsql/plpython3.so
+ %endif
+
  %if %test
  %files test
  %defattr(-,postgres,postgres)