Thread: Extensions Dependency Checking
Hackers, I wanted to get a (ok, not so) quick note in about this before the beta dropped. I've been thinking about the "requires"parameter on extensions control files. Right now it just lists the names of extensions that are required for theextension to run: requires = 'foo, bar' But I'm assuming that at some point there's going to be something a bit more robust: specifically, requiring a minimum version,perhaps something like: requires = 'foo 1.0, bar 0.31.4' I know that Dimitri has thought about this a bit, and I suspect we can all agree that most extensible systems have some wayto at least specify the minimum version of a dependency. Is this something that's likely to come down the pike, perhapsin 9.2 or later? If so, we return to the issue of comparing version strings in order to resolve/verify dependencies. One way to solve the problem would be either for a developer to specify a version order in the control file, or else to ensurethat the upgrade scripts are there for every single version. But frankly, I would oppose something like this, as it once again puts the onus on the extension developer to take care ofwhat is essentially a bookkeeping detail. And if I'm any example, there will often be mistakes, typos, etc. I think a better solution would be to mandate a version string format for extensions *today*. This would remove the onuson the developer, while making version comparisons inside the database efficient and predictable. Now, I know that ideas has been rejected in the past (at least for *right now*), but I'm here to tell you that not solvingthis problem up front is fodder for a nightmare later. Perl, for example, has only recently standardized on a versionnumbering scheme, but thanks to a 15 year history of none at all, it has to bend over backwards to try to manage anyversion scheme in existence. I mean, just look at this mess: http://www.dagolden.com/index.php/369/version-numbers-should-be-boring/ I definitely think that version numbers should be boring. A few simple rules and then forget about it, it should just work. I certainly didn't want to deal with this issue in PGXN, where dependency checking is planned for the command-line client.So I decided to use semantic versions. They have now been implemented as a native type in PostgreSQL, thanks to SamVilain porting the PL/pgSQL-based domain I originally wrote. The code is here: http://master.pgxn.org/dist/semver/0.2.0/ The reason I went with semantic versions, rather than just a numeric, as I'd originally planned, is because they are almostexactly the same format as PostgreSQL version numbers. So they were useful for specifying a dependency on a particularversion of PostgreSQL. So by letting the core project set the standard, with just a minor modification of thatstandard, I get something that just works but is much more flexible than a numeric version. FWIW, the only differencethat I've noticed between semantic versions and PostgreSQL versions is in developer/alpha/beta releases. So wherePostgreSQL has released 9.1alpha5, the semantic version is 9.1.0alpha5. (And the semver data type can convert from theformer to the latter). So, to summarize: * I think we're going to need a formal version string spec for extensions. * If that's true, I think it should be specified *now*, before extensions are in the wild, so that we don't end up with thelegacy version string nightmares that Perl modules suffer from. * If we do adopt a spec, I think it should reflect the PostgreSQL core version strings as closely as possible, and shouldbe fully compatible with them. * So it might be worth looking at semver or something similar to integrate. Thoughts? Best, David
On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david@kineticode.com> wrote: > But I'm assuming that at some point there's going to be something a bit more robust: specifically, requiring a minimumversion, perhaps something like: > > requires = 'foo 1.0, bar 0.31.4' Or maybe: requires = 'foo = 1.0, bar >= 0.31.4' > * I think we're going to need a formal version string spec for extensions. I agree. > * If that's true, I think it should be specified *now*, before extensions are in the wild, so that we don't end up withthe legacy version string nightmares that Perl modules suffer from. I think I agree with this, too. > * If we do adopt a spec, I think it should reflect the PostgreSQL core version strings as closely as possible, and shouldbe fully compatible with them. I am less sure about this one. > * So it might be worth looking at semver or something similar to integrate. No. It's too late to be monkeying with this. I think for 9.1 we will need to content ourselves with setting a good precedent, rather than enforcing it programatically. It's not going to work to insist on all numeric version strings anyway, because we've already got this 'FROM unpackaged' bit floating around. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david@kineticode.com> wrote:
>> * I think we're going to need a formal version string spec for extensions.
> I agree.
I don't. We deliberately decided *not* to have any wired-in
interpretation of extension numbers, and I don't think that decision
needs to be reversed. David can choose to enforce something for stuff
distributed through PGXN if he wishes, but that's no concern of the core
server's. In particular I'm really skeptical of the theory that we need
or should want version restrictions in Requires references. The
equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
and I see no reason why we'd need it more than they do.
>> * So it might be worth looking at semver or something similar to integrate.
> No. It's too late to be monkeying with this. I think for 9.1 we will
> need to content ourselves with setting a good precedent, rather than
> enforcing it programatically. It's not going to work to insist on all
> numeric version strings anyway, because we've already got this 'FROM
> unpackaged' bit floating around.
Once 9.1 is out, it'll probably be too late to dictate any semantics for
version numbers, because somebody will have done something incompatible
with it before 9.2 is released. If we are going to try to insist on
this, now is the time. But I don't agree with that position.
regards, tom lane
On Mon, Apr 4, 2011 at 5:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Fri, Apr 1, 2011 at 11:45 AM, David E. Wheeler <david@kineticode.com> wrote: >>> * I think we're going to need a formal version string spec for extensions. > >> I agree. > > I don't. We deliberately decided *not* to have any wired-in > interpretation of extension numbers, and I don't think that decision > needs to be reversed. David can choose to enforce something for stuff > distributed through PGXN if he wishes, but that's no concern of the core > server's. In particular I'm really skeptical of the theory that we need > or should want version restrictions in Requires references. The > equivalent feature in RPM is deprecated for Fedora/RedHat packaging use, > and I see no reason why we'd need it more than they do. Oh, really? How can you possibly get by without it? Dependencies of this type are all over the place. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Apr 4, 2011, at 2:48 PM, Tom Lane wrote: > Once 9.1 is out, it'll probably be too late to dictate any semantics for > version numbers, because somebody will have done something incompatible > with it before 9.2 is released. If we are going to try to insist on > this, now is the time. Yes, exactly my point. If it's going to be done, it should be now. > But I don't agree with that position. Why not? Best. David
On Mon, Apr 4, 2011 at 6:06 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> I don't. We deliberately decided *not* to have any wired-in >> interpretation of extension numbers, and I don't think that decision >> needs to be reversed. David can choose to enforce something for stuff >> distributed through PGXN if he wishes, but that's no concern of the core >> server's. In particular I'm really skeptical of the theory that we need >> or should want version restrictions in Requires references. The >> equivalent feature in RPM is deprecated for Fedora/RedHat packaging use, >> and I see no reason why we'd need it more than they do. > > Oh, really? How can you possibly get by without it? Dependencies of > this type are all over the place. I think the general movement is toward *feature* dependancies. So for intstance, an extension can specify what *feature* it requires, and difference "versions" of an extension can provide different "features". In that case, you don't really need extenson "foo" > 2.1, you need the feature that foo 2.1.x provides, maybe "foo-api-2" (note that 2 would be part of a "name", not any comparison aware version. I'm already going to be "naming" my extensions with "major versions" as part of the name (like all the distro postgresql packages) so my versions will only ever be simple integers of exactly compatable objects. But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html, I don't see any "provides" mechanism. That might be something actually needed if we are trying to avoid "version" comparisons and want to be describing actual dependencies... -- Aidan Van Dyk Create like a god, aidan@highrise.ca command like a king, http://www.highrise.ca/ work like a slave.
Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Apr 4, 2011 at 5:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> ... In particular I'm really skeptical of the theory that we need
>> or should want version restrictions in Requires references. �The
>> equivalent feature in RPM is deprecated for Fedora/RedHat packaging use,
>> and I see no reason why we'd need it more than they do.
> Oh, really? How can you possibly get by without it? Dependencies of
> this type are all over the place.
Yeah, but the proposed solution doesn't actually fix the problem.
There are various issues, but the fundamental objection is that when you
really need it the information is usually wrong/obsolete, especially if
it's maintained by hand. As an example, if we have foo depending on
bar and it says "Requires: bar >= 2.0", that's all well and good, but
it tells us nothing whatsoever about whether this version of foo will
work with bar 3.0, especially if 3.0 postdates the time when the foo
author put that into his specfile. Maybe we should really interpret
that as "Requires: bar >= 2.0 && < 3.0", or maybe not. It's also very
easy for foo's author to start using some nice_bar_function() without
realizing that that was actually only added in bar 2.1.
If you want automatic dependency checking to actually do something
useful, you need something very substantially finer-grained and more
complex than this. In the Fedora world they're mostly depending on
per-shared-library sonames and per-exported-symbol symbol versioning,
and even then it doesn't always do what you wished. But I'll note
that neither of those mechanisms bothers to infer any version ordering:
either you have a match, or you don't.
The equivalent thing of what's now considered good packaging practice
would probably be to extract lists of specific objects provided or
required by a module, preferably automatically. The requires side of
that might be doable just by inspecting pg_depend. The provides side
would be harder, mainly because you'd want some way to omit objects
that are considered purely internal to a module.
Anyway, I'm uneager to adopt a practice that was discredited years ago
among real packaging systems. I think it'll just delay somebody
deciding to fix it properly.
regards, tom lane
Aidan Van Dyk <aidan@highrise.ca> writes: > On Mon, Apr 4, 2011 at 6:06 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> Oh, really? �How can you possibly get by without it? �Dependencies of >> this type are all over the place. > I think the general movement is toward *feature* dependancies. So for > intstance, an extension can specify what *feature* it requires, and > difference "versions" of an extension can provide different > "features". Right. > But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html, > I don't see any "provides" mechanism. Yes, some sort of manual Provides: (in addition to automatically extracted Provides:) would likely be part of any serious solution. We're not there yet, and we're not going to get there in time for 9.1. But in any case, mechanisms that involve version ordering comparisons seem to be on their way out for deciding whether package A is compatible with package B. regards, tom lane
Aidan Van Dyk <aidan@highrise.ca> writes: > I think the general movement is toward *feature* dependancies. So for > intstance, an extension can specify what *feature* it requires, and > difference "versions" of an extension can provide different > "features". That sounds like what Emacs is doing too. > But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html, > I don't see any "provides" mechanism. That might be something > actually needed if we are trying to avoid "version" comparisons and > want to be describing actual dependencies... The 'provides' mechanism can be added later I think. It's like saying that in 9.1 an extension 'foo' provides the feature 'foo' and you can't control that, whereas in future version you will be able to control what your extension (and its specific version) provides. Also we will be able to list what the PostgreSQL server provides, maybe, so that compile time options can be depended on by extensions. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Apr 4, 2011, at 3:57 PM, Tom Lane wrote: >> I think the general movement is toward *feature* dependancies. So for >> intstance, an extension can specify what *feature* it requires, and >> difference "versions" of an extension can provide different >> "features". > > Right. Sounds like a book-keeping nightmare for extension developers. It will discourage large or rapidly-evolving extensions likepgTAP because it will be a PITA to specify features. >> But checking http://developer.postgresql.org/pgdocs/postgres/extend-extensions.html, >> I don't see any "provides" mechanism. > > Yes, some sort of manual Provides: (in addition to automatically > extracted Provides:) would likely be part of any serious solution. <shed type="bike">I'd like to request "Features:" instead of "Provides:".</shed> > We're not there yet, and we're not going to get there in time for 9.1. > But in any case, mechanisms that involve version ordering comparisons > seem to be on their way out for deciding whether package A is > compatible with package B. This is news to me, frankly, and the bookkeeping requirements seem potentially awful. If it's possible that it won't work out this way, that those arguing for version dependency resolution end up getting theconsensus, not having a version string format is going to be a nightmare. On the other hand, if we added one now, andfeature dependency tracking won the day, well, a version string format could always be loosened later. Best, David
On Tue, Apr 5, 2011 at 4:20 PM, David E. Wheeler <david@kineticode.com> wrote:
> On Apr 4, 2011, at 3:57 PM, Tom Lane wrote:
>
>>> I think the general movement is toward *feature* dependancies. So for
>>> intstance, an extension can specify what *feature* it requires, and
>>> difference "versions" of an extension can provide different
>>> "features".
>>
>> Right.
>
> Sounds like a book-keeping nightmare for extension developers. It will discourage large or rapidly-evolving
extensionslike pgTAP because it will be a PITA to specify features.
Sure, but if you want, the "feature" you can provide can be something like: pgtap-1.0 (or any of
pgtap-0.2{0,1,2,3,4}).
And if your package is backwards compatable, it could even provide: pgtap-0.25 pgtap-0.24 pgtap-0.23
And that also means that you don't have to screw every body over when
some future pgtap-123.45 is no longer compatible, and the extensions
have relied on "$VERSION > 0.23" meaning they'll work with it.
I mean, PG itself is an example. Does pg > 8.4 mean your code will
work with all future (or even past, but > 8.4) PG versions?
>> We're not there yet, and we're not going to get there in time for 9.1.
>> But in any case, mechanisms that involve version ordering comparisons
>> seem to be on their way out for deciding whether package A is
>> compatible with package B.
>
> This is news to me, frankly, and the bookkeeping requirements seem potentially awful.
>
> If it's possible that it won't work out this way, that those arguing for version dependency resolution end up getting
theconsensus, not having a version string format is going to be a nightmare. On the other hand, if we added one now,
andfeature dependency tracking won the day, well, a version string format could always be loosened later.
As someone who has had to try and deal with "package versions" for
dependencies in RPM and DEB, and been through the hell that is open
source package variants, all with the ability to turn on/off features
at configure/compile time, a just versions even with <, <=, =, >=, >
all mapped correctly isn't good enough.
Of course, I'ld love for extension in 9.1 to provide a basic
provides/features for my extension to give, but if that train has
already left the station, I don't have much choice ;-(
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.
On Apr 5, 2011, at 1:42 PM, Aidan Van Dyk wrote:
> Sure, but if you want, the "feature" you can provide can be something like:
> pgtap-1.0 (or any of pgtap-0.2{0,1,2,3,4}).
>
> And if your package is backwards compatable, it could even provide:
> pgtap-0.25
> pgtap-0.24
> pgtap-0.23
I see, I get it.
> And that also means that you don't have to screw every body over when
> some future pgtap-123.45 is no longer compatible, and the extensions
> have relied on "$VERSION > 0.23" meaning they'll work with it.
I see.
> I mean, PG itself is an example. Does pg > 8.4 mean your code will
> work with all future (or even past, but > 8.4) PG versions?
I see. So the extension author can more easily tell users when compatibility has been dropped for something. That makes
sense.
> As someone who has had to try and deal with "package versions" for
> dependencies in RPM and DEB, and been through the hell that is open
> source package variants, all with the ability to turn on/off features
> at configure/compile time, a just versions even with <, <=, =, >=, >
> all mapped correctly isn't good enough.
Yeah. The use of an implicit >= in CPAN modules has been a decent 90% solution, but it does cause headaches for people
thatthey can't express things better. The ability to do so would require a mini-language with more operators,
precedence,grouping, etc.
> Of course, I'ld love for extension in 9.1 to provide a basic
> provides/features for my extension to give, but if that train has
> already left the station, I don't have much choice ;-(
Yeah, but the way it is doesn't break the ability to do it later. I suspect that Dim and Tom will be thinking about it
for9.2.
Anyway, your post helps me to understand things better, and so I'm less insistent about imposing a version numbering
schemenow (though I still think it would be more useful to have one than not).
Best,
David
On Tue, Apr 5, 2011 at 4:51 PM, David E. Wheeler <david@kineticode.com> wrote: >> Of course, I'ld love for extension in 9.1 to provide a basic >> provides/features for my extension to give, but if that train has >> already left the station, I don't have much choice ;-( > > Yeah, but the way it is doesn't break the ability to do it later. I suspect that Dim and Tom will be thinking about itfor 9.2. > > Anyway, your post helps me to understand things better, and so I'm less insistent about imposing a version numbering schemenow (though I still think it would be more useful to have one than not). Versions are useful for figuring out if I should upgrade packages or not. But I believe the extension framework has explicitly made the "upgrade" problem a manual one at this point, either taking destination versions from the control, or the alter command. So for PGXN's problem, I see the point of versions being required. But for installation the dependancy graph, "provides/features" rather than versions are much more useful. And automatic feature/provides (like library so, and symbol versions in the OS package world, "objects" in PG world) would definitely be nice, but my Makefile can build those for me for now until 9.2 (or 9.3, 9.3, etc), if only I had a way to track them with my installed extension ;-) </stop begging> a. -- Aidan Van Dyk Create like a god, aidan@highrise.ca command like a king, http://www.highrise.ca/ work like a slave.
On Apr 5, 2011, at 1:59 PM, Aidan Van Dyk wrote: > Versions are useful for figuring out if I should upgrade packages or > not. But I believe the extension framework has explicitly made the > "upgrade" problem a manual one at this point, either taking > destination versions from the control, or the alter command. > > So for PGXN's problem, I see the point of versions being required. Okay, thanks, I'm convinced (and relieved; I hated to be bringing something like this up so close to launch). > But for installation the dependancy graph, "provides/features" rather > than versions are much more useful. And automatic feature/provides > (like library so, and symbol versions in the OS package world, > "objects" in PG world) would definitely be nice, but my Makefile can > build those for me for now until 9.2 (or 9.3, 9.3, etc), if only I had > a way to track them with my installed extension ;-) </stop begging> I'm sure a patch would be welcomed. ;-P Best, David