Thread: Confusion over Python drivers
I saw this on reddit and thought I might drop a line.<br /><br />I went through this same issue, trying to find a postgresqldriver. Mostly, I had no intention of using psycopg because of its copyleft licensing. (Of course, I don't needto go into why.)<br /><br />Anyways, here's some info that might help on three alternatives:<br />Py-postgresql, PyGreSQL,and PG3000<br /><br />Py-postgresql is active (but I only see 1 person committing)<br />MIT/BSD style license<br/> It uses some some C for speed.<br /> It has DB-API and PG-API bindings.<br /><br />Problem: is it Python 3 only!<br/><br /><br />PyGreSQL (looking to not be very active in the last few months)<br />MIT/BSD style license<br />I'mvague on it's features, whether it has C optimizations, or if it is a version limited to Python 3 or 2.x<br /><br /><br/>PG8000 is active at the moment<br />MIT/BSD style license<br />It works on 2.x and Python 3<br /><br />Problem: itis pure Python, so it is slower<br /><br /><br />If you plan on pursuing this, I'd recommend talking to people on on 3projects. Here's what I would describe as ideal in a project:<br /><br />* MIT/BSD style license<br />* works on Python2.x and 3<br />* C optimizations for speed (or whatever for speed)<br /><br />------------------<br />That's that part. Now for some problems that would need solving in order for the db api to be extremely useful/userfriendly in the Pythonworld:<br />------------------<br />* SQLAlchemy bindings/support -- as of right now only psycopyg works in SQLAlchemywithout problems. PG8000 will work, but you need an experimental version of boht PG8000 and SQLAlchemy. The othertwo do not work. SQLAlchemy support is probably most needed of anything. SQLAlchemy support should not only be possiblebut default.<br />* Django bindings (these are unique to Django and nobody but psycopg has them, which is not good). Django support should not only be possible but default.<br /> * Needs 2.x and Python 3 compatability<br />* It isinevitable that if the library uses C optimizations, then that means they need to provide lots of different binaries --think 32bit vs 64bit and then 2.4, 2.5, 3.0 etc.... Some combinations will often be left out making it hard for someoneto use/install it.<br /><br /><br /><br />Of course all of this is from the perspective of Python users. Of course,you have your own features that you want from your end (from PostgreSQL's perspective). Perhaps this info would helpyou to know which avenue to pursue.<br /><br /><hr />Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. <ahref="http://clk.atdmt.com/GBL/go/201469229/direct/01/" target="_new">Sign up now.</a>
Kevin, > Of course all of this is from the perspective of Python users. Of > course, you have your own features that you want from your end (from > PostgreSQL's perspective). Perhaps this info would help you to know > which avenue to pursue. That's invaluable. Thanks for chiming in! --Josh Berkus
kevinar18@hotmail.com (Kevin Ar18) writes:
> Of course all of this is from the perspective of Python users. Of
> course, you have your own features that you want from your end (from
> PostgreSQL's perspective). Perhaps this info would help you to know
> which avenue to pursue.
No, those seem like fine ways of getting a good perspective on the
issue.
I happen not to use Python much, so there's a certain aspect of "don't
care" on my part... but that doesn't imply that my "PostgreSQL
perspective" would tend to override yours. Instead, I think that the
"Python users' perspective" *is* a mighty important thing.
The interface needs aspects of "cleanness" on both sides of the
interface...
- On the Python side, it needs to "play well" in a variety of ways that you seem to have described nicely, some
technical,some licensing oriented. Some relating to interfacing to further bits of Python and to applications and
frameworkswritten in Python.
- On the PostgreSQL side, there's certainly a preference for "licensing simplicity."
Note that most of the issues there really lie on the Python side, which
underlines the importance of "Python users' perspective."
Further, the "ideal" and "issues/problems" that you point out all seem
reasonable. The good seems good and the bad seems like things that do
indeed need to be accepted as consequences of the good.
It will doubtless help guide assistance.
--
output = reverse("moc.liamg" "@" "enworbbc")
"...as a robotics designer once told me, you don't really appreciate
how smart a moron is until you try to design a robot..."
-- Jerry Pournelle
I hope people don't mind my asking about this on the list... as I hinted at before, I don't really follow the developmentof PostgreSQL, I was just interested in the Python driver project that I heard about.<br /><br />Anyways, as Iunderstand it, the current goal is to use psycopg and get it changed to LGPL (assuming all the contributors of psycopg agreeand confirm they did not use GPL code from any other location). Is this correct?<br /><br /><br />When I first heardabout the endeavor, I thought the goal was to take one or several of the non-copyleft projects, which were rather unfocused,and work with those teams to produce a really good implementation for Python. However, as I understand it (basedon what Greg told me) the license is not really an issue as long as it is not GPL; instead, the PostgreSQL team wouldmostly prefer something that is nearly done, so as to have to do much more work. Is this a correct assessment?<br /><br/><br />Based on that, I guess my question is what would it have taken to have picked one of BSD/MIT projects and workingwith those people instead? In other words, what key things affected the decision for psycopg? What areas is it sofar ahead in or that would have just been too much work to fix in the other implementations?<br /><br /><br />Anyways,I hope this message doesn't come across as bad form. It's unfortunate for me that there was not a good enoughBSD/MIT project; but I can live without right? :) Still, I just thought I might ask and find out a little more aboutwhat the team was looking for in a PostgreSQL implementation, and maybe do a little research myself (to see if anythingwas missed).<br /><br /><hr />Hotmail: Free, trusted and rich email service. <a href="http://clk.atdmt.com/GBL/go/201469228/direct/01/"target="_new">Get it now.</a>
Kevin Ar18 <kevinar18@hotmail.com> writes:
> When I first heard about the endeavor, I thought the goal was to take
> one or several of the non-copyleft projects, which were rather
> unfocused, and work with those teams to produce a really good
> implementation for Python. However, as I understand it (based on what
> Greg told me) the license is not really an issue as long as it is not
> GPL; instead, the PostgreSQL team would mostly prefer something that
> is nearly done, so as to have to do much more work. Is this a correct
> assessment?
Well, all else being equal we'd certainly prefer a library that was
licensed more like the core Postgres database. However, we don't have
infinite resources, and an LGPL license is not a showstopper (at least
not to the people who seem to be willing to work on this problem).
The attractiveness of the license has to be balanced against how much
work we'd have to put in and how long it will take to get results.
Not being a python user myself, I wasn't paying all that close attention
to the discussion, but that's my sense of how the decision went.
If you feel that a BSD/MIT license is a must-have for your purposes,
you're certainly free to push development of one of the other driver
projects instead, and to try to organize some other people to help.
I don't believe anyone is trying to funnel all development effort into
psycopg2.
regards, tom lane
> Well, all else being equal we'd certainly prefer a library that was<br />> licensed more like the core Postgresdatabase. However, we don't have<br />> infinite resources, and an LGPL license is not a showstopper (at least<br/>> not to the people who seem to be willing to work on this problem).<br />> The attractiveness of the licensehas to be balanced against how much<br />> work we'd have to put in and how long it will take to get results.<br/>> <br />> Not being a python user myself, I wasn't paying all that close attention<br />> to the discussion,but that's my sense of how the decision went.<br />> <br />> If you feel that a BSD/MIT license is a must-havefor your purposes,<br />> you're certainly free to push development of one of the other driver<br />> projectsinstead, and to try to organize some other people to help.<br />> I don't believe anyone is trying to funnel alldevelopment effort into<br />> psycopg2.<br />Thanks for the reply.<br /><br />I guess that's good advice; I supposeI should just do that and talk to some of the teams about it. It would probably help a lot to focus on just one implementationinstead of several, even if it's not the same one as what the PostgreSQL team works on. :)<br /><br /><hr />Hotmail:Trusted email with Microsoft’s powerful SPAM protection. <a href="http://clk.atdmt.com/GBL/go/201469226/direct/01/"target="_new">Sign up now.</a>
Kevin Ar18 wrote: > > Based on that, I guess my question is what would it have taken to have > picked one of BSD/MIT projects and working with those people instead? > In other words, what key things affected the decision for psycopg? > What areas is it so far ahead in or that would have just been too much > work to fix in the other implementations? A lot of it as I see it is plain old fashioned popularity resulting from long sustained development. psycopg has been around since 2001, the current version is already compatible with SQLAlchemy, Django, Zope, PyDO, and it's integral to the large Python/PostgreSQL toolchain from Skype including tools like Londiste. When something is supported in that many places, and the main complaints are its license and documentation rather than its code, I know I'm not going to start by assuming I should just hack on somebody else's code to try and replace it just because their license is a little better. And I'd consider BSD/MIT only a little better than the LGPL. That sort of bad decision making is how we got to so many abandoned Python drivers in the first place. If everybody who decided they were going to write their own from scratch had decided to work on carefully and painfully refactoring and improving PyGreSQL instead, in an incremental way that grew its existing community along the way, we might have a BSD driver with enough features and users to be a valid competitor to psycopg2. But writing something shiny new from scratch is fun, while worrying about backward compatibility and implementing all the messy parts you need to really be complete on a project like this isn't, so instead we have a stack of not quite right drivers without any broad application support. (Aside: I have a bit of vocabulary I regularly use for this now. Open-source projects that just ignore working on an existing stack of working implementations, to instead start from scratch to build something of questionably improved fanciness instead regardless of its impact on backward compatibility and the existing userbase, have in my terminology "PulseAudioed" the older projects). The gauntlet I would throw down for anyone who thinks there's a better approach here is to demonstrate a driver that has working support going back to Python 2.4 for any two of the apps on my list above. Get even that much of a foothold, and maybe the fact that yours is more Pythonic, cleaner, or better licensed matters. Until then, working apps have to be the primary motivation for what to work on here, unless there's a really terrible problem with the driver. The existing psycopg license and the web site issues were in combination enough to reach that level of total issues for a number of people. With the news from Federico that a license improvement is approaching and signs of major documentation improvements, that problem seems like it will soon be solved as far as I'm concerned. When someone manages to make their alternative driver a prerequisite for an app I need, only at that point will that driver show back up on my radar. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
Tom Lane wrote: > If you feel that a BSD/MIT license is a must-have for your purposes, > you're certainly free to push development of one of the other driver > projects instead, and to try to organize some other people to help. > I don't believe anyone is trying to funnel all development effort into > psycopg2. > Certainly not, and I hope no one has gotten the impression that there's anything "official" being recognized about psycopg happening here because it hasn't. Anointing a "one true driver" (a phrase that seems to keep popping up in external discussions of this topic) isn't the sort of thing the PostgreSQL core does. And all that happens to people who ignore what I tell them to do is that they receive a steady stream of long, sarcastic e-mails for a while afterwards. I just updated http://wiki.postgresql.org/wiki/Python_PostgreSQL_Driver_TODO to reflect a few corrections I noted while researching (everybody else is welcome to edit that page too you know), and to include some of the recent feedback showing up on this list. I know I was just looking for a Python driver that is compatible with the apps I most often run into, documented well enough that I can write my own if I feel like it, fast enough, and free enough that I can deploy the result wherever I want. That seemed similar to the priorities other people who had an opinion here suggested too. Pragmatically, psycopg2 just seemed to have the shortest path toward being something useful to the largest userbase in that sort of context, and we've unofficially rolled down that path a bit. This rabble-rousing seems to have nudged both development communities toward being more closely aligned in the future in a couple of ways, which is great, but I wouldn't read much more into things than that. Other projects will continue to grow and shrink, and the pure Python ones in particular continue to be quite valuable because they fill a niche that psycopg2 doesn't target at all. I'd sure like to see all three of those projects merge into one big one though. My bet has to be on pg8000, since it has the perfect license, supports all the necessary Python versions, and it's been around long enough (almost two years) that support for it is already in the latest SQLAlchemy beta. We seem to have revitalized discussion around modernizing PyGreSQL too, so I wouldn't discount that one completely yet either. For those who feel a true BSD license is vital, I direct you toward http://mailman.vex.net/pipermail/pygresql/2010-February/002315.html to learn more about what direction they could use some help in going. But whatever you do, don't start another project instead. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
On Wed, 2010-02-10 at 23:13 -0500, Greg Smith wrote: > Until then, working apps have to > be the primary motivation for what to work on here, unless there's a > really terrible problem with the driver. The existing psycopg license > and the web site issues were in combination enough to reach that level > of total issues for a number of people. With the news from Federico > that a license improvement is approaching and signs of major > documentation improvements, that problem seems like it will soon be > solved as far as I'm concerned. When someone manages to make their > alternative driver a prerequisite for an app I need, only at that point > will that driver show back up on my radar. Keep in mind that backwards compatibility is not the only issue here; forwards compatibility matters as well*. A lot of the encoding issues I wrote up ( http://wiki.postgresql.org/wiki/Driver_development ) will probably be real bugs in a python3 application using a driver that doesn't understand encoding properly. Obviously this is less urgent than having a driver that works now, but it's still important. I think we would attract some goodwill from the python community if we were helping them move to python3, rather than sitting around waiting 'til they've already moved and decided that they can't use postgresql. Regards,Jeff Davis * I tried installing psycopg2-2.0.13 and the build system apparently doesn't support python3.1, so I assume that psycopg2 doesn't support python3 at all.
>Obviously this is less urgent than having a driver that works now, but >it's still important. I think we would attract some goodwill from the >python community if we were helping them move to python3, rather than >sitting around waiting 'til they've already moved and decided that they >can't use postgresql. It's very, very difficult (but not impossible) to support both python 2 and 3 simultaneously, particularly if you have non-trivial C extension code. Even the python gods will admit that it's still early days. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/
On 12/02/2010 01:00, Jeff Davis wrote:
> * I tried installing psycopg2-2.0.13 and the build system apparently
> doesn't support python3.1, so I assume that psycopg2 doesn't support
> python3 at all.
python3 was almost completely supported some months ago but then I had
to fix some bugs and almost 99% of psycopg users are still with python2
so the python2 branch is in a better shape. But most of the work is done
so, after the next release, I'll start porting changes to the python3
branch ("master" on git) and finish the work.
federico
--
Federico Di Gregorio fog@initd.orgViva il pollo! Viva il pollo! --
sisterconfusionContinuoa preferire il coniglio -- vodka
On Fri, 2010-02-12 at 10:38 +0100, Federico Di Gregorio wrote:
> On 12/02/2010 01:00, Jeff Davis wrote:
> > * I tried installing psycopg2-2.0.13 and the build system apparently
> > doesn't support python3.1, so I assume that psycopg2 doesn't support
> > python3 at all.
>
> python3 was almost completely supported some months ago but then I had
> to fix some bugs and almost 99% of psycopg users are still with python2
> so the python2 branch is in a better shape. But most of the work is done
> so, after the next release, I'll start porting changes to the python3
> branch ("master" on git) and finish the work.
Ok, sounds great.
Do you attempt to handle the encoding issues when working with python3?
I'd be interested to see your approach, because I didn't get much
feedback on the doc I wrote.
Regards,Jeff Davis
Jeff Davis wrote: > Keep in mind that backwards compatibility is not the only issue here; > forwards compatibility matters as well*. A lot of the encoding issues I > wrote up ( http://wiki.postgresql.org/wiki/Driver_development ) will > probably be real bugs in a python3 application using a driver that > doesn't understand encoding properly. > Sure, but the benefit of being the de-facto standard to some problem, because you're already supported by everybody, is that it's easier to attract the most help to move forward too. The disclaimers on investments always read "past performance is not indicative of future results". I don't think that's really true in the open-source project case. Generally, I'd rather trust a project that's been keeping up with things for a long time already to continue to do so, particularly one that's built up a community around it already, rather than risk switching to one without a known good track record in that record. (Exercise for the reader: consider how what I just said applies in both a positive and negative way toward MySQL and its associated development model) -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com