Thread: Re: [GENERAL] programmatic way to fetch latest release for a given major.minor version

On Tue, Apr 10, 2007 at 08:36:10AM +0100, Dave Page wrote:
> Andrew Hammond wrote:
> > On 4/9/07, CAJ CAJ <pguser@gmail.com> wrote:
> >> On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
> >> <andrew.george.hammond@gmail.com> wrote:
> >> > I'm writing a script that wants to know the latest release for a given
> >> > major.minor version. Is there some better way than parsing
> >> > http://www.postgresql.org/ftp/source/ or trying to
> >> connect to ftp
> >> > (which is invariably timing out on me today. Is that box getting
> >> > hammered or something?) and doing the parsing that? Both approaches
> >> > feel quite awkward to me.
> >>
> >> Use wget to download via  HTTP (added recently).  Probably wise to add a
> >> couple mirrors in your script.
> >
> > I'm not asking how to download stuff. I'm asking how to figure out the
> > current release number for a given major.minor. I thought that was
> > clear in my original post, but I guess not. For example, how do I
> > determine (programmatically) the lastest version of 8.1.
>
> Currently you can't without parsing pages.

I've been meaning to do this for a while now, and I guess this might be a
good motivator ;-) The idea would, of course, be to build this from a set of
common data with the frontpage of the website, so they're always in sync.
As I see it, there are basically three ways of doing it:

1) Fold it into mirrors.xml. That's going to be a bit hard given how the
schema is (since the root element is mirror), but it would work. Another
downside is that is that people who just want to check the latest version
will have to pull down the whole mirror list file. This file is also
rewritten often, so checks on If-Modified-Since will fail.

2) Create a new file with a specific schema. Something like:
<version>
 <version major="8.2" minor="8.2.3" />
 <version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

3) Use RSS format. Not exactly sure how it would look, but it can certainly
be done in RSS...


I think I prefer option 2, but what do others think?

//Magnus

Magnus Hagander wrote:
> 2) Create a new file with a specific schema. Something like:
> <version>
>  <version major="8.2" minor="8.2.3" />
>  <version major="8.1" minor="8.1.8" />
> </version>
> This is the most lightweight solution.

More like:

<versions>
  <version major="8" minor="2" revision="3" />
  <version major="8" minor="1" revision="8" />
</versions>

But I can't help thinking that we should have some additional values for
release notes, download sub-URLs (to be appended to the mirror roots) etc.

Regards, Dave.

    I love Open Source XD

    http://ethan.tira-thompson.com/cvslog2web/

    Note that this is overkill (but it would look SEXY on the site).
    However, the original poster probably wants to know when to update his
servers, so he won't care about CVS commits...
    If there was a RSS feed of new postgres versions I'd subscribe to it
though.

> <versions>
>   <version major="8" minor="2" revision="3" />
>   <version major="8" minor="1" revision="8" />
> </versions>
>
> But I can't help thinking that we should have some additional values for
> release notes, download sub-URLs (to be appended to the mirror roots)
> etc.

On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:
> Magnus Hagander wrote:
> > 2) Create a new file with a specific schema. Something like:
> > <version>
> >  <version major="8.2" minor="8.2.3" />
> >  <version major="8.1" minor="8.1.8" />
> > </version>
> > This is the most lightweight solution.
>
> More like:
>
> <versions>
>   <version major="8" minor="2" revision="3" />
>   <version major="8" minor="1" revision="8" />
> </versions>

But that doesn't reflect our terminology. Per our own terminology, bioth
8.1 and 8.2 are major versions. Not only is that what we've been saying for
years, we even documented it at
http://www.postgresql.org/support/versioning.


> But I can't help thinking that we should have some additional values for
> release notes, download sub-URLs (to be appended to the mirror roots) etc.

Yeah, thus the X in XML :-)
But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

//Magnus

On Tue, Apr 10, 2007 at 11:09:50AM +0200, Listmail wrote:
>
>     I love Open Source XD
>
>     http://ethan.tira-thompson.com/cvslog2web/

Hmm. Don't tell people about my secret plans :)

(Though I hadn't looked at that piece of software in particylar)

>     Note that this is overkill (but it would look SEXY on the site).
>     However, the original poster probably wants to know when to update
>     his  servers, so he won't care about CVS commits...
>     If there was a RSS feed of new postgres versions I'd subscribe to it
> though.

That's the argument for RSS - you acn subscribe to it from standard RSS
readers.
Perhaps we need to do both...

//Magnus

Magnus Hagander wrote:
> On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:
>> Magnus Hagander wrote:
>>> 2) Create a new file with a specific schema. Something like:
>>> <version>
>>>  <version major="8.2" minor="8.2.3" />
>>>  <version major="8.1" minor="8.1.8" />
>>> </version>
>>> This is the most lightweight solution.
>> More like:
>>
>> <versions>
>>   <version major="8" minor="2" revision="3" />
>>   <version major="8" minor="1" revision="8" />
>> </versions>
>
> But that doesn't reflect our terminology. Per our own terminology, bioth
> 8.1 and 8.2 are major versions. Not only is that what we've been saying for
> years, we even documented it at
> http://www.postgresql.org/support/versioning.

Yeah yeah, but terminology aside, having 2 or three digits in each
attribute is just wrong!

>> But I can't help thinking that we should have some additional values for
>> release notes, download sub-URLs (to be appended to the mirror roots) etc.
>
> Yeah, thus the X in XML :-)

:-p

> But given that we might want to add things like this, having our own custom
> XML format certainly makes a bit more sense, it might be harder to try to
> trick it into RSS.

Agreed.

/D

On Tue, Apr 10, 2007 at 12:03:44PM +0100, Dave Page wrote:
> Magnus Hagander wrote:
> > On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:
> >> Magnus Hagander wrote:
> >>> 2) Create a new file with a specific schema. Something like:
> >>> <version>
> >>>  <version major="8.2" minor="8.2.3" />
> >>>  <version major="8.1" minor="8.1.8" />
> >>> </version>
> >>> This is the most lightweight solution.
> >> More like:
> >>
> >> <versions>
> >>   <version major="8" minor="2" revision="3" />
> >>   <version major="8" minor="1" revision="8" />
> >> </versions>
> >
> > But that doesn't reflect our terminology. Per our own terminology, bioth
> > 8.1 and 8.2 are major versions. Not only is that what we've been saying for
> > years, we even documented it at
> > http://www.postgresql.org/support/versioning.
>
> Yeah yeah, but terminology aside, having 2 or three digits in each
> attribute is just wrong!

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

//Magnus

Magnus Hagander wrote:
> Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
> sense to say you're on version 8, in the given context, so why should the
> XML data pretend there is?

Because serving the data in the decomposed format gives the consumer the
maximum flexibility to do as they wish with the data. I find it hard to
see why, as a relational database guy, you'd want to offer the data as
"8.1", "8.1.3" etc. when you can just give the three parts separately
and allow people to check whatever they need without having to chop up
strings!

Imagine wanting to display only the details of the 8.x releases on a
site for example. In your schema, you'd have to use a substring match on
an attribute value to filter out 6.x and 7.x.

Regards,D ave.

On Tue, Apr 10, 2007 at 12:35:38PM +0100, Dave Page wrote:
> Magnus Hagander wrote:
> > Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
> > sense to say you're on version 8, in the given context, so why should the
> > XML data pretend there is?
>
> Because serving the data in the decomposed format gives the consumer the
> maximum flexibility to do as they wish with the data. I find it hard to
> see why, as a relational database guy, you'd want to offer the data as
> "8.1", "8.1.3" etc. when you can just give the three parts separately
> and allow people to check whatever they need without having to chop up
> strings!
>
> Imagine wanting to display only the details of the 8.x releases on a
> site for example. In your schema, you'd have to use a substring match on
> an attribute value to filter out 6.x and 7.x.

That is actually precisely my point. It makes *no sense* to filter based on
8.x. 8.0 is no more a major release than 7.4. And we'd encourage people to
do that.

//Magnus

On Tue, Apr 10, 2007 at 02:44:42PM +0200, Martijn van Oosterhout wrote:
> On Tue, Apr 10, 2007 at 12:18:52PM +0200, Magnus Hagander wrote:
> > But given that we might want to add things like this, having our own custom
> > XML format certainly makes a bit more sense, it might be harder to try to
> > trick it into RSS.
>
> I'd say do it in the format you're most comfortable with. Then some
> XSLT wizard can convert it into RSS from there...

Too late, I already did the RSS. Which was very simple, because of the RSS
framework stuff already in place on the website.

*However*, that one doesn't contain all the information that a program
would want, and certainly not in an easily parseable format for them. It's
designed for people in the other end. So I intend to do a "proper XML
format" as well.

//Magnus

Am Dienstag, 10. April 2007 13:35 schrieb Dave Page:
> Imagine wanting to display only the details of the 8.x releases on a
> site for example. In your schema, you'd have to use a substring match on
> an attribute value to filter out 6.x and 7.x.

That use case is just as valid as wanting to show only releases >= 7.4.  The
user could run an arithmetic expression to check for that and >= 8.0.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

On Tue, Apr 10, 2007 at 12:18:52PM +0200, Magnus Hagander wrote:
> But given that we might want to add things like this, having our own custom
> XML format certainly makes a bit more sense, it might be harder to try to
> trick it into RSS.

I'd say do it in the format you're most comfortable with. Then some
XSLT wizard can convert it into RSS from there...

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Magnus Hagander wrote:
> That is actually precisely my point. It makes *no sense* to filter based on
> 8.x. 8.0 is no more a major release than 7.4.

Yes it is - that's precisely why it was 8.0 and not 7.5.

/D

Dave Page escribió:
> Magnus Hagander wrote:
> > That is actually precisely my point. It makes *no sense* to filter based on
> > 8.x. 8.0 is no more a major release than 7.4.
>
> Yes it is - that's precisely why it was 8.0 and not 7.5.

That was merely a marketing artifact; it was called 7.5 until the very
end of the devel cycle.

--
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Dave Page escribió:
>> Magnus Hagander wrote:
>>> That is actually precisely my point. It makes *no sense* to filter based on
>>> 8.x. 8.0 is no more a major release than 7.4.
>> Yes it is - that's precisely why it was 8.0 and not 7.5.
>
> That was merely a marketing artifact; it was called 7.5 until the very
> end of the devel cycle.
>

Yes, but marketing is one reason why someone might want to group 8.x,
7.x etc on a website which is exactly the sort of thing this code is for.

As others have said, yes, you could do it but looking at a substring of
the version, and yes, you could do it with mathematical comparisons on
major.minor (with limitations - what happens if we get to 8.10 ?), but
would we suggest people use those techniques for searching their
databases for matching records, or would we suggest storing the
interesting elements in different columns for ease of use, flexibility
and efficiency? How does this differ (aside from the obvious)?

Regards, Dave

Dave Page escribió:
> Alvaro Herrera wrote:
> > Dave Page escribió:
> >> Magnus Hagander wrote:
> >>> That is actually precisely my point. It makes *no sense* to filter based on
> >>> 8.x. 8.0 is no more a major release than 7.4.
> >> Yes it is - that's precisely why it was 8.0 and not 7.5.
> >
> > That was merely a marketing artifact; it was called 7.5 until the very
> > end of the devel cycle.
>
> Yes, but marketing is one reason why someone might want to group 8.x,
> 7.x etc on a website which is exactly the sort of thing this code is for.

Ah, but then it's not an decision to be made on arithmetics alone -- you
have to build a higher-level semantic comparison.  Because if you want
to group by something else, for example the quality of Windows support,
you surely don't want 8.0 nor 8.1, because they have unfixable problems
(the pgstat bug, autovacuum not working).  You need to include only 8.2
and higher.

> As others have said, yes, you could do it but looking at a substring of
> the version, and yes, you could do it with mathematical comparisons on
> major.minor (with limitations - what happens if we get to 8.10 ?), but
> would we suggest people use those techniques for searching their
> databases for matching records, or would we suggest storing the
> interesting elements in different columns for ease of use, flexibility
> and efficiency? How does this differ (aside from the obvious)?

It makes sense to store things separately when they have a semantic
difference.  What we call "major" is the first two digits and dot.  We
call "minor" to the third digit, and that's all.  We don't have
"revisions".  This is how it has ever been and we even document it as
such.  Offering the first two digits separately would be a mistake
because it causes confusion over what's significant -- the first digit
by itself is not significant.

--
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:
> It makes sense to store things separately when they have a semantic
> difference.  What we call "major" is the first two digits and dot.  We
> call "minor" to the third digit, and that's all.  We don't have
> "revisions".  This is how it has ever been and we even document it as
> such.  Offering the first two digits separately would be a mistake
> because it causes confusion over what's significant -- the first digit
> by itself is not significant.

While I agree with this position, ISTM that an easy compromise is
available: put both formats into the data.

            regards, tom lane

Alvaro Herrera wrote:
> It makes sense to store things separately when they have a semantic
> difference.  What we call "major" is the first two digits and dot.  We
> call "minor" to the third digit, and that's all.  We don't have
> "revisions".  This is how it has ever been and we even document it as
> such.  Offering the first two digits separately would be a mistake
> because it causes confusion over what's significant -- the first digit
> by itself is not significant.

Yes, and for the most part it's true (certainly as far as the purpose of
that documentation is concerned), however it all falls apart next time
we have the 'are there enough new features in this release to bump the
first digit' discussion which, along with the fact that we have bumped
it twice now, is proof in itself that we do give it some meaning.

Anyhoo, I've said my piece now - I'll await the inevitable replies
telling me I'm a donut and then move on :-)

Regards, Dave

On Tue, Apr 10, 2007 at 12:18:52PM +0200, Magnus Hagander wrote:
> On Tue, Apr 10, 2007 at 09:52:52AM +0100, Dave Page wrote:
> > Magnus Hagander wrote:
> > > 2) Create a new file with a specific schema. Something like:
> > > <version>
> > >  <version major="8.2" minor="8.2.3" />
> > >  <version major="8.1" minor="8.1.8" />
> > > </version>
> > > This is the most lightweight solution.
> >
> > More like:
> >
> > <versions>
> >   <version major="8" minor="2" revision="3" />
> >   <version major="8" minor="1" revision="8" />
> > </versions>
>

Ok, I've added one of these as well now, alongside the RSS feed. You can
get the pg specific XML file at:
http://www.postgresql.org/versions.xml

If someone wants the schema change, react *now*. Later on we can only
append to it, and not change it :)

//Magnus

> If someone wants the schema change, react *now*. Later on we can only
> append to it, and not change it :)

    Since I like to complain...

<numericversion v1="8" v2="2" v3="3"/>

    Suppose you someday add another dot, or a "b" for beta, wouldn't it be
better to have

<versionlist><item>8</item><item>2</item><item>3</item></versionlist>

    ... or <item value="3" />

On Wed, Apr 11, 2007 at 06:49:18PM +0200, Listmail wrote:
>
>
> >If someone wants the schema change, react *now*. Later on we can only
> >append to it, and not change it :)
>
>     Since I like to complain...
>
> <numericversion v1="8" v2="2" v3="3"/>
>
>     Suppose you someday add another dot, or a "b" for beta, wouldn't it
>     be  better to have
>
> <versionlist><item>8</item><item>2</item><item>3</item></versionlist>

IIRC, but not entirely sure, the order of items in XML is not guaranteed.
So you'd need something like
<versionlist><item seq="1">8</item><item seq="2">2</item> etc etc

I'm not sure, but I have some kind of memory of that ;-)


As for beta, we're only going to be listing production versions in this
one. It's there to list the latest available version in each released
series.
And if we add another dot, we can just add a v4="7" attribute. Adding is
not a problem, only modifying.

//Magnus