Thread: Seeking Insights on Roll-out Process for Multiple PyPI Packages
Dear Psycopg mailing list members,
I'm reaching out to inquire about the roll-out process for managing two
PyPI packages from a single source repository,
similar to the setup employed by the Psycopg Python package (psycopg2
and psycopg2-binary).
As I work on a project with a comparable requirement, I would greatly
appreciate any insights the Psycopg team can offer in this regard.
Specifically, I'm interested in the following:
1) How did the Psycopg team handle deploying and distributing two
PyPI packages from a single source repository?
2) Were there any challenges encountered during the roll-out
process, and if so, how were they addressed?
3) Are there any recommended tools, scripts, or configuration
adjustments for effectively managing the two PyPI packages?
Thank you for your time and consideration. If you need more information
or have any questions, I'm happy to answer them.
Best regards,
Jan-Hendrik Lorenz
--
Dr. Jan-Hendrik Lorenz * Software Consultant *
jan-hendrik.lorenz@tngtech.com
TNG Technology Consulting GmbH, Beta-Str. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Thomas Endres
Sitz: Unterföhring * Amtsgericht München * HRB 135082
Attachment
Hello,
On Fri, 2 Jun 2023, 16:25 Jan-Hendrik Lorenz, <jan-hendrik.lorenz@tngtech.com> wrote:
1) How did the Psycopg team handle deploying and distributing two
PyPI packages from a single source repository?
With one line of sed: https://github.com/psycopg/psycopg2/blob/8b17e218bea6dbaf43f27d6aa97bafb33e2fb66f/scripts/build/wheel_linux_before_all.sh#L48
The package name is replaced in the setup.py, and that's enough to create a different distribution package from the same Python module.
2) Were there any challenges encountered during the roll-out
process, and if so, how were they addressed?
The biggest problem is faced with dependency tracking, because pypi metadata don't actually consider the possibility of different distribution package providing the same python package. Both psycopg2 and psycopg2-binary can be installed in the same environment, but they will trample on each other's files, with undefined results. Libraries shouldn't depend on -binary, but of course many either don't read the indication or they consider themselves superior and do it otherwise, causing problems when two libraries both depending on psycopg2 are installed alongside.
In Psycopg 3 we are doing something similar, but we solved large part of the problems. There is a pure python package defining the interface, which can be used as dependency target, and optional speedup. The psycopg_c speedup requires to be built on the client, so it's similar to psycopg2; the psycopg_binary speedup is precompiled and distributed as wheel together with the library dependencies, so it's similar to psycopg2-binary. The two distribution packages install two distinct python packages, so they can be installed alongside. See https://www.psycopg.org/psycopg3/docs/basic/install.html
In this case too, both packages are created from the same source code; modifications are a bit more than sed'ing a single file but also not difficult: see the script https://github.com/psycopg/psycopg/blob/master/tools/build/copy_to_binary.py
3) Are there any recommended tools, scripts, or configuration
adjustments for effectively managing the two PyPI packages?
Feel free to use the scripts and techniques above for your own use. I strongly advise against distributing the same python package in different distribution packages and suggest you to follow the approach of psycopg 3.
Best
-- Daniele