Re: Seeking Insights on Roll-out Process for Multiple PyPI Packages - Mailing list psycopg

From Daniele Varrazzo
Subject Re: Seeking Insights on Roll-out Process for Multiple PyPI Packages
Date
Msg-id CA+mi_8ZCLTjCoio4Z_5PSzk_8RQdq1qgvZgykT_P=Pxc6ERc6A@mail.gmail.com
Whole thread Raw
In response to Seeking Insights on Roll-out Process for Multiple PyPI Packages  (Jan-Hendrik Lorenz <jan-hendrik.lorenz@tngtech.com>)
List psycopg
Hello,

On Fri, 2 Jun 2023, 16:25 Jan-Hendrik Lorenz, <jan-hendrik.lorenz@tngtech.com> wrote:

     1) How did the Psycopg team handle deploying and distributing two
PyPI packages from a single source repository?


The package name is replaced in the setup.py, and that's enough to create a different distribution package from the same Python module.


     2) Were there any challenges encountered during the roll-out
process, and if so, how were they addressed?

The biggest problem is faced with dependency tracking, because pypi metadata don't actually consider the possibility of different distribution package providing the same python package. Both psycopg2 and psycopg2-binary can be installed in the same environment, but they will trample on each other's files, with undefined results. Libraries shouldn't depend on -binary, but of course many either don't read the indication or they consider themselves superior and do it otherwise, causing problems when two libraries both depending on psycopg2 are installed alongside.

In Psycopg 3 we are doing something similar, but we solved large part of the problems. There is a pure python package defining the interface, which can be used as dependency target, and optional speedup. The psycopg_c speedup requires to be built on the client, so it's similar to psycopg2; the psycopg_binary speedup is precompiled and distributed as wheel together with the library dependencies, so it's similar to psycopg2-binary. The two distribution packages install two distinct python packages, so they can be installed alongside. See https://www.psycopg.org/psycopg3/docs/basic/install.html

In this case too, both packages are created from the same source code; modifications are a bit more than sed'ing a single file but also not difficult: see the script https://github.com/psycopg/psycopg/blob/master/tools/build/copy_to_binary.py

     3) Are there any recommended tools, scripts, or configuration
adjustments for effectively managing the two PyPI packages?

Feel free to use the scripts and techniques above for your own use. I strongly advise against distributing the same python package in different distribution packages and suggest you to follow the approach of psycopg 3.

Best

-- Daniele 

psycopg by date:

Previous
From: Jan-Hendrik Lorenz
Date:
Subject: Seeking Insights on Roll-out Process for Multiple PyPI Packages
Next
From: David Raymond
Date:
Subject: conn.read_only not honored in autocommit mode