Thread: Supporting plpython 2+3 builds better

I wrote:
> After reading the recent thread about python 2 vs python 3 support,
> I thought I'd amuse myself by trying to get plpython3 supported in
> the Fedora packages.  That turned out to be unreasonably painful
> (which is something we oughta fix eventually), but it worked,

To give you an idea of what "unreasonably painful" means, attached is
the specfile diff needed to make this happen.  I will not comment on
the fragility of this beyond observing that the "touch -r" commands
are *necessary*, else you get incorrect results.

I think we really need to do something to make this type of build less
painful, because as far as I can see most distros are going to be
wanting to support both python 2 and 3 for awhile to come.

A sketch of a solution might go like this:

* Extend configure to have two switches, say "--with-python" and
"--with-python3", which you can select either or both of.  If you
want to pick executables that are not named "python" and "python3",
perhaps write it like "--with-python3=/usr/bin/python3.2mu".  (I'd
be inclined to remove the dependency on a PYTHON envvar altogether.)

* Make configure output two independent sets of variables into
Makefile.global, viz

    python2_includespec    = ...
    python2_libdir        = ...
    python2_libspec        = ...
    python2_additional_libs    = ...
    python2_configdir    = ...
    python2_majorversion    = ...
    python2_version        = ...

    python3_includespec    = ...
    python3_libdir        = ...
    python3_libspec        = ...
    python3_additional_libs    = ...
    python3_configdir    = ...
    python3_majorversion    = ...
    python3_version        = ...

* Make plpython's Makefile build, test, install plpython2 and/or
plpython3 depending on what's set in Makefile.global.

I'm not sure yet whether it's possible to do this without duplicating a
lot of logic in config/python.m4 and plpython's Makefile.

Another problem is that Makefile.shlib isn't designed to build more than
one shared library per directory, which means that we might end up
having to copy all the source files into a new plpython3 subdirectory
anyway.  We're already doing some of that with the regression test
files, though.

Thoughts?

            regards, tom lane

diff --git a/postgresql.spec b/postgresql.spec
index 524d81c..62ce352 100644
*** a/postgresql.spec
--- b/postgresql.spec
***************
*** 35,40 ****
--- 35,41 ----
  %{!?test:%global test 1}
  %{!?upgrade:%global upgrade 1}
  %{!?plpython:%global plpython 1}
+ %{!?plpython3:%global plpython3 1}
  %{!?pltcl:%global pltcl 1}
  %{!?plperl:%global plperl 1}
  %{!?ssl:%global ssl 1}
*************** BuildRequires: systemd-units
*** 107,112 ****
--- 108,117 ----
  BuildRequires: python-devel
  %endif

+ %if %plpython3
+ BuildRequires: python3-devel
+ %endif
+
  %if %pltcl
  BuildRequires: tcl-devel
  %endif
*************** Install this if you want to write databa
*** 265,278 ****

  %if %plpython
  %package plpython
! Summary: The Python procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}

  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python.
  %endif

  %if %pltcl
--- 270,295 ----

  %if %plpython
  %package plpython
! Summary: The Python2 procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}

  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 2.
! %endif
!
! %if %plpython3
! %package plpython3
! Summary: The Python3 procedural language for PostgreSQL
! Group: Applications/Databases
! Requires: %{name}-server%{?_isa} = %{version}-%{release}
!
! %description plpython3
! The postgresql-plpython3 package contains the PL/Python3 procedural language,
! which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 3.
  %endif

  %if %pltcl
*************** CFLAGS="$CFLAGS -DLINUX_OOM_SCORE_ADJ=0"
*** 346,351 ****
--- 363,447 ----
  # CFLAGS=`echo $CFLAGS| sed -e "s|-O2|-O1|g" `
  # %%endif

+ # plpython requires separate configure/build runs to build against python 2
+ # versus python 3.  Our strategy is to do the python 3 run first, then make
+ # distclean and do it again for the "normal" build.  Note that the installed
+ # Makefile.global will reflect the python 2 build, which seems appropriate
+ # since that's still considered the default plpython version.
+ %if %plpython3
+
+ export PYTHON=/usr/bin/python3
+
+ # These configure options must match main build
+ %configure --disable-rpath \
+ %if %beta
+     --enable-debug \
+     --enable-cassert \
+ %endif
+ %if %plperl
+     --with-perl \
+ %endif
+ %if %pltcl
+     --with-tcl \
+     --with-tclconfig=%{_libdir} \
+ %endif
+ %if %plpython3
+     --with-python \
+ %endif
+ %if %ldap
+     --with-ldap \
+ %endif
+ %if %ssl
+     --with-openssl \
+ %endif
+ %if %pam
+     --with-pam \
+ %endif
+ %if %kerberos
+     --with-krb5 \
+     --with-gssapi \
+ %endif
+ %if %uuid
+     --with-ossp-uuid \
+ %endif
+ %if %xml
+     --with-libxml \
+     --with-libxslt \
+ %endif
+ %if %nls
+     --enable-nls \
+ %endif
+ %if %sdt
+     --enable-dtrace \
+ %endif
+ %if %selinux
+     --with-selinux \
+ %endif
+     --with-system-tzdata=/usr/share/zoneinfo \
+     --datadir=/usr/share/pgsql
+
+ # Fortunately we don't need to build much except plpython itself
+ cd src/backend
+ make submake-errcodes
+ cd ../..
+ cd src/pl/plpython
+ make %{?_smp_mflags} all
+ cd ..
+ # save built form in a directory that "make distclean" won't touch
+ cp -a plpython plpython3
+ cd ../..
+
+ # must also save this version of Makefile.global for later
+ cp src/Makefile.global src/Makefile.global.python3
+
+ make distclean
+
+ %endif
+
+ unset PYTHON
+
+ # Normal (not python3) build begins here
+
  %configure --disable-rpath \
  %if %beta
      --enable-debug \
*************** rm -f src/tutorial/GNUmakefile
*** 409,414 ****
--- 505,528 ----
      pushd src/pl
      make MAX_CONNECTIONS=5 check
      popd
+ %if %plpython3
+     # must install Makefile.global that selects python3
+     mv src/Makefile.global src/Makefile.global.save
+     cp src/Makefile.global.python3 src/Makefile.global
+     touch -r src/Makefile.global.save src/Makefile.global
+     # because "make check" does "make install" on the whole tree,
+     # we must temporarily install plpython3 as src/pl/plpython,
+     # since that is the subdirectory src/pl/Makefile knows about
+     mv src/pl/plpython src/pl/plpython2
+     mv src/pl/plpython3 src/pl/plpython
+     pushd src/pl/plpython
+     make MAX_CONNECTIONS=5 check
+     popd
+     # and clean up our mess
+     mv src/pl/plpython src/pl/plpython3
+     mv src/pl/plpython2 src/pl/plpython
+     mv -f src/Makefile.global.save src/Makefile.global
+ %endif
      pushd contrib
      make MAX_CONNECTIONS=5 check
      popd
*************** rm -f src/tutorial/GNUmakefile
*** 442,447 ****
--- 556,571 ----

  make DESTDIR=$RPM_BUILD_ROOT install-world

+ %if %plpython3
+     mv src/Makefile.global src/Makefile.global.save
+     cp src/Makefile.global.python3 src/Makefile.global
+     touch -r src/Makefile.global.save src/Makefile.global
+     pushd src/pl/plpython3
+     make DESTDIR=$RPM_BUILD_ROOT install
+     popd
+     mv -f src/Makefile.global.save src/Makefile.global
+ %endif
+
  # multilib header hack; note pg_config.h is installed in two places!
  # we only apply this to known Red Hat multilib arches, per bug #177564
  case `uname -i` in
*************** cp /dev/null devel.lst
*** 588,593 ****
--- 713,719 ----
  cp /dev/null plperl.lst
  cp /dev/null pltcl.lst
  cp /dev/null plpython.lst
+ cp /dev/null plpython3.lst

  %if %nls
  %find_lang ecpg-%{majorversion}
*************** cat plpgsql-%{majorversion}.lang >>serve
*** 622,627 ****
--- 748,758 ----
  %find_lang plpython-%{majorversion}
  cat plpython-%{majorversion}.lang >>plpython.lst
  %endif
+ %if %plpython3
+ # plpython3 shares message files with plpython
+ %find_lang plpython-%{majorversion}
+ cat plpython-%{majorversion}.lang >>plpython3.lst
+ %endif
  %if %pltcl
  %find_lang pltcl-%{majorversion}
  cat pltcl-%{majorversion}.lang >>pltcl.lst
*************** fi
*** 907,916 ****

  %if %plpython
  %files plpython -f plpython.lst
! %{_datadir}/pgsql/extension/plpython*
  %{_libdir}/pgsql/plpython2.so
  %endif

  %if %test
  %files test
  %defattr(-,postgres,postgres)
--- 1051,1067 ----

  %if %plpython
  %files plpython -f plpython.lst
! %{_datadir}/pgsql/extension/plpythonu*
! %{_datadir}/pgsql/extension/plpython2*
  %{_libdir}/pgsql/plpython2.so
  %endif

+ %if %plpython3
+ %files plpython3 -f plpython3.lst
+ %{_datadir}/pgsql/extension/plpython3*
+ %{_libdir}/pgsql/plpython3.so
+ %endif
+
  %if %test
  %files test
  %defattr(-,postgres,postgres)

On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
> To give you an idea of what "unreasonably painful" means, attached is
> the specfile diff needed to make this happen.  I will not comment on
> the fragility of this beyond observing that the "touch -r" commands
> are *necessary*, else you get incorrect results.

I think an easier way is to configure two vpath builds and install the
pieces you want from each one. yo

> Another problem is that Makefile.shlib isn't designed to build more
> than one shared library per directory,

That's the main problem, but fixing it would be very useful in other
places as well.  I had it on my radar to do something about that.

>  which means that we might end up having to copy all the source files
> into a new plpython3 subdirectory anyway.  We're already doing some of
> that with the regression test files, though.

Doing it with the test files is already annoying enough.  For instance,
we don't have a static list of all the files we need to copy (because of
expected file variants that are not tracked in the makefiles), so we
need to copy the files again on each run.  If you extend this to all
source files, things would get totally bizarre.

A less invasive method might be to set up a second directory
pl/plpython3 which vpath-builds from pl/plpython.

But frankly, I'd like to work on the multiple-shlibs-in-one-directory
issue, because all of the above issues and more are projected in their
own strange ways on something like the transforms feature.  For example,
how would you organize the source code for an extension module that
provides a new data type and adapters for plpython{2,3}u and plperl{,u},
with tests and all that?

Peter Eisentraut <peter_e@gmx.net> writes:
> On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
>> To give you an idea of what "unreasonably painful" means, attached is
>> the specfile diff needed to make this happen.  I will not comment on
>> the fragility of this beyond observing that the "touch -r" commands
>> are *necessary*, else you get incorrect results.

> I think an easier way is to configure two vpath builds and install the
> pieces you want from each one.

I thought about that, and didn't like it, because then you have no proof
that the plpython3 you built in the one tree will actually play with the
core code you built in the other tree.  Messy as my solution is, the
regression test step does actually test the intended live combination of
executables.

>> Another problem is that Makefile.shlib isn't designed to build more
>> than one shared library per directory,

> That's the main problem, but fixing it would be very useful in other
> places as well.  I had it on my radar to do something about that.

This would be a good thing.  Got any ideas how to do it?
        regards, tom lane

On Sun, 2012-09-09 at 03:35 -0400, Tom Lane wrote:
> >> Another problem is that Makefile.shlib isn't designed to build more
> >> than one shared library per directory,
>
> > That's the main problem, but fixing it would be very useful in other
> > places as well.  I had it on my radar to do something about that.
>
> This would be a good thing.  Got any ideas how to do it?

Here is a very rough patch.  It obviously will need a lot of
fine-tuning, but it's the idea.

Excerpts from Peter Eisentraut's message of lun sep 10 09:50:42 -0300 2012:
> On Sun, 2012-09-09 at 03:35 -0400, Tom Lane wrote:
> > >> Another problem is that Makefile.shlib isn't designed to build more
> > >> than one shared library per directory,
> >
> > > That's the main problem, but fixing it would be very useful in other
> > > places as well.  I had it on my radar to do something about that.
> >
> > This would be a good thing.  Got any ideas how to do it?
>
> Here is a very rough patch.  It obviously will need a lot of
> fine-tuning, but it's the idea.

I remember trying to do this for the mb/conversion_procs subdir years
ago, to make them build in parallel to save some time.  It didn't go
anywhere but the basic idea seems similar in spirit.  Maybe we can use
this there too to make it fast.

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 9/10/12 9:26 AM, Alvaro Herrera wrote:
> I remember trying to do this for the mb/conversion_procs subdir years
> ago, to make them build in parallel to save some time.  It didn't go
> anywhere but the basic idea seems similar in spirit.  Maybe we can use
> this there too to make it fast.

Parallel builds across subdirectories should work now, so that shouldn't
be necessary anymore.  Although removing some directory depth might be
nice, but there is a general hesitation about renaming files.