Thread: Some download statistics

Some download statistics

From
"Magnus Hagander"
Date:
(crossposting this to hackers, I'm sure there are interested people
there as well)

Since Dave set the site up for tracking clickthroughs, I hit the db with
a couple of queries to count our downloads. This is what I came up with:

I did some simple pivoting in Excel and split it into categories win32,
source, sig (MD5 or PGP signatures), RPMs, split (the split tarballs),
pgadmin and ODBC. Other stuff was so little that I cut it.

The numbers for just-past-release may be off a bit because of the move
of wwwmaster. Not sure what happened to the db.

Just before and about 24 hours after the release of 8.0, we had the
following:
win32        16480
source    5223
sig        2016
rpm        1908
split        1689
pgadmin    370
odbc        261
Grand Total    27947


Total tally since the tracking started (2005-01-11):
win32        106878    56,37%
source    39058        20,60%
rpm        11703        6,17%
split        9135        4,82%
sig        8931        4,71%
pgadmin    7281        3,84%
odbc        6601        3,48%
Total     189587


And the versions being downloaded (total):
8.0.1        85511
8.0.0        82517
notpg        14526        (pgadmin, odbc etc)
7.4.6        4297
7.4.7        2736
Grand Total    189587


This does *not* include bittorrent downloads.

Some conclusions that I draw from this:
1) The win32 distribution is very popular. Some of this is no doubt
because it's new. Also, about 3500 of those downloads are RC5 downloads.
RC5 downloads of the source were during this time so few that they
didn't show up after my cutoff.

2) Most of the people who use RPMs on their linux systems wait for their
distribution to ship the RPMs and don't get it. Or they are alraedy set
up to download directly from their mirror and not from the website (the
tracker only counts downloads that were initiated from the website)

3) There doesnt' seem to be much point to the distribution splits. A
total of less than 5% the *number* of downloads. And most people
probably get more than one file, so in reality that number shuold
proably be divided by 4 or 5.
I know several people who downloaded source *plus* the split ones,
because "hey, I need postgresql. And I certainly need base too. And I
need docs.". They don't realise it's included in the main tarball.
Frankly, I'd suggest dropping the splits. Thoughts?

4) People upgraded to 8.0.1. Not a lot of people have upgraded their 7.4
systems.


As always, it's statistics. You can say anything you want with it...
Someone who chews numbers for a living might be able to get more out of
it. If someone is interested in the excel pivot tables with this stuff
in it, let me know. But it's nothing exciting there :)


//Magnus

Re: Some download statistics

From
"Marc G. Fournier"
Date:
On Thu, 24 Feb 2005, Magnus Hagander wrote:

> 3) There doesnt' seem to be much point to the distribution splits. A
> total of less than 5% the *number* of downloads. And most people
> probably get more than one file, so in reality that number shuold
> proably be divided by 4 or 5.
> I know several people who downloaded source *plus* the split ones,
> because "hey, I need postgresql. And I certainly need base too. And I
> need docs.". They don't realise it's included in the main tarball.
> Frankly, I'd suggest dropping the splits. Thoughts?

this tracks only those going in through the web ... this doesn't track
those using an ftp client going in and downloading files ... specifically,
I know that all of the FreeBSD ports are based on the split distributions
... in the case of the postgresql80-server port, it only downloads base
and opt and ignores the rest ...


----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Some download statistics

From
John DeSoi
Date:
Hi Magnus,

On Feb 24, 2005, at 11:35 AM, Magnus Hagander wrote:

> I did some simple pivoting in Excel and split it into categories win32,
> source, sig (MD5 or PGP signatures), RPMs, split (the split tarballs),
> pgadmin and ODBC. Other stuff was so little that I cut it.


Assuming this is from a HTTP log, I'm just curious if you took into
account that many Windows users have "download managers" that make
repeated requests to get a single file. You see this in the log with a
206 (partial content) HTTP response code. So you might have anywhere
from 10 to 20 hits in the log to download a single file. This makes it
a pain to get accurate download statistics.


John DeSoi, Ph.D.
http://pgedit.com/
Power Tools for PostgreSQL


Re: Some download statistics

From
Troels Arvin
Date:
On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:

> I know several people who downloaded source *plus* the split ones,
> because "hey, I need postgresql. And I certainly need base too. And I
> need docs.". They don't realise it's included in the main tarball.
> Frankly, I'd suggest dropping the splits. Thoughts?

I also found the split sources + a non-split sources version to be
confusing. As you, I think that splitting should be dropped.

-- 
Greetings from Troels Arvin, Copenhagen, Denmark




Re: Some download statistics

From
Tom Lane
Date:
Troels Arvin <troels@arvin.dk> writes:
> On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:
>> Frankly, I'd suggest dropping the splits. Thoughts?

> I also found the split sources + a non-split sources version to be
> confusing. As you, I think that splitting should be dropped.

Perhaps the confusion issue could be addressed by keeping the split
sources in a separate subdirectory:
pub/source/v.8.0.1/    postgresql-8.0.1.tar.bz2    postgresql-8.0.1.tar.bz2.md5    postgresql-8.0.1.tar.gz
postgresql-8.0.1.tar.gz.md5   split-tarballs/        postgresql-base-8.0.1.tar.bz2
postgresql-base-8.0.1.tar.bz2.md5       etc...
 

I suspect the demand for the split tarballs is mighty low nowaways,
but it's probably not zero yet.

Also: I notice that the README file that's supposed to tell people about
the split-tarball scheme is not present in any of the recent-version
subdirectories, so it's no wonder that they are confused.
        regards, tom lane


Re: Some download statistics

From
"Marc G. Fournier"
Date:
On Thu, 24 Feb 2005, Tom Lane wrote:

> Troels Arvin <troels@arvin.dk> writes:
>> On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:
>>> Frankly, I'd suggest dropping the splits. Thoughts?
>
>> I also found the split sources + a non-split sources version to be
>> confusing. As you, I think that splitting should be dropped.
>
> Perhaps the confusion issue could be addressed by keeping the split
> sources in a separate subdirectory:
>
>     pub/source/v.8.0.1/
>         postgresql-8.0.1.tar.bz2
>         postgresql-8.0.1.tar.bz2.md5
>         postgresql-8.0.1.tar.gz
>         postgresql-8.0.1.tar.gz.md5
>         split-tarballs/
>             postgresql-base-8.0.1.tar.bz2
>             postgresql-base-8.0.1.tar.bz2.md5
>             etc...

That sounds like a reasonable compromise ... I could do that right away, 
if nobody is in disagreement ... ?

> I suspect the demand for the split tarballs is mighty low nowaways,
> but it's probably not zero yet.

Considering that every new FreeBSD install uses them exclusively, I don't 
expect it to drop to zero in the near future ..

> Also: I notice that the README file that's supposed to tell people about
> the split-tarball scheme is not present in any of the recent-version
> subdirectories, so it's no wonder that they are confused.

Actually, its never  been ... just  checked, its in the root directory ... 
but, with that in mind, I've added putting a copy in there to the release 
script ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664


Re: Some download statistics

From
Ernst Herzberg
Date:
On Thursday 24 February 2005 21:43, Marc G. Fournier wrote:
[.......]
> >
> >     pub/source/v.8.0.1/
> >         postgresql-8.0.1.tar.bz2
> >         postgresql-8.0.1.tar.bz2.md5
> >         postgresql-8.0.1.tar.gz
> >         postgresql-8.0.1.tar.gz.md5
> >         split-tarballs/
> >             postgresql-base-8.0.1.tar.bz2
> >             postgresql-base-8.0.1.tar.bz2.md5
> >             etc...
>
> That sounds like a reasonable compromise ... I could do that right away,
> if nobody is in disagreement ... ?

Hm, there is a distribution that uses the split tarballs: gentoo.

snipped from /usr/portage/dev-db/postgresql/postgresql-8.0.1-r1.ebuild :

SRC_URI="mirror://postgresql/source/v${PV}/${PN}-base-${MY_PV}.tar.bz2
mirror://postgresql/source/v${PV}/${PN}-opt-${MY_PV}.tar.bz2  doc? (
mirror://postgresql/source/v${PV}/${PN}-docs-${MY_PV}.tar.bz2)"
 

Better is to drop a 'bug' to bugs.gentoo.org so that the ebuild will be 
fixed;-)


<Earny>


Re: Some download statistics

From
"Magnus Hagander"
Date:
>>>> Frankly, I'd suggest dropping the splits. Thoughts?
>>
>>> I also found the split sources + a non-split sources version to be
>>> confusing. As you, I think that splitting should be dropped.
>>
>> Perhaps the confusion issue could be addressed by keeping the split
>> sources in a separate subdirectory:
>>
>>     pub/source/v.8.0.1/
>>         postgresql-8.0.1.tar.bz2
>>         postgresql-8.0.1.tar.bz2.md5
>>         postgresql-8.0.1.tar.gz
>>         postgresql-8.0.1.tar.gz.md5
>>         split-tarballs/
>>             postgresql-base-8.0.1.tar.bz2
>>             postgresql-base-8.0.1.tar.bz2.md5
>>             etc...
>
>That sounds like a reasonable compromise ... I could do that
>right away,
>if nobody is in disagreement ... ?

Seems reasonable to me.


>> I suspect the demand for the split tarballs is mighty low nowaways,
>> but it's probably not zero yet.
>
>Considering that every new FreeBSD install uses them
>exclusively, I don't
>expect it to drop to zero in the near future ..

Do they pull fromt eh pg mirrors, or from their own? I realise they
still need it, of course, but it'd be interesting to know.


//Magnus


Re: Some download statistics

From
"Dave Page"
Date:

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of John DeSoi
> Sent: 24 February 2005 19:20
> To: Magnus Hagander
> Cc: pgsql-hackers@postgresql.org; pgsql-www@postgresql.org
> Subject: Re: [HACKERS] Some download statistics
>
> Hi Magnus,
>
> On Feb 24, 2005, at 11:35 AM, Magnus Hagander wrote:
>
> > I did some simple pivoting in Excel and split it into
> categories win32,
> > source, sig (MD5 or PGP signatures), RPMs, split (the split
> tarballs),
> > pgadmin and ODBC. Other stuff was so little that I cut it.
>
>
> Assuming this is from a HTTP log,

It's not. When you click on a mirror flag after selecting a file under
http://www.postgresql.org/ftp/ the click is logged and the client
redirected to the requested file on the selected mirror.

/D

Re: Some download statistics

From
Tom Lane
Date:
"Marc G. Fournier" <scrappy@postgresql.org> writes:
> On Thu, 24 Feb 2005, Tom Lane wrote:
>> Also: I notice that the README file that's supposed to tell people about
>> the split-tarball scheme is not present in any of the recent-version
>> subdirectories, so it's no wonder that they are confused.

> Actually, its never  been ... just  checked, its in the root directory ... 
> but, with that in mind, I've added putting a copy in there to the release 
> script ...

It is present in the subdirectories for some older releases, eg v7.3.
        regards, tom lane


Re: Some download statistics

From
"Marc G. Fournier"
Date:
On Thu, 24 Feb 2005, Magnus Hagander wrote:

>>>>> Frankly, I'd suggest dropping the splits. Thoughts?
>>>
>>>> I also found the split sources + a non-split sources version to be
>>>> confusing. As you, I think that splitting should be dropped.
>>>
>>> Perhaps the confusion issue could be addressed by keeping the split
>>> sources in a separate subdirectory:
>>>
>>>     pub/source/v.8.0.1/
>>>         postgresql-8.0.1.tar.bz2
>>>         postgresql-8.0.1.tar.bz2.md5
>>>         postgresql-8.0.1.tar.gz
>>>         postgresql-8.0.1.tar.gz.md5
>>>         split-tarballs/
>>>             postgresql-base-8.0.1.tar.bz2
>>>             postgresql-base-8.0.1.tar.bz2.md5
>>>             etc...
>>
>> That sounds like a reasonable compromise ... I could do that
>> right away,
>> if nobody is in disagreement ... ?
>
> Seems reasonable to me.
>
>
>>> I suspect the demand for the split tarballs is mighty low nowaways,
>>> but it's probably not zero yet.
>>
>> Considering that every new FreeBSD install uses them
>> exclusively, I don't
>> expect it to drop to zero in the near future ..
>
> Do they pull fromt eh pg mirrors, or from their own? I realise they
> still need it, of course, but it'd be interesting to know.

MASTER_SITES=   ${MASTER_SITE_PGSQL}

which expands to:

MASTER_SITE_PGSQL+= \        ftp://ftp3.us.postgresql.org/pub/postgresql/%SUBDIR%/ \
ftp://ftp8.us.postgresql.org/postgresql/%SUBDIR%/\        ftp://ftp9.us.postgresql.org/pub/mirrors/postgresql/%SUBDIR%/
\       ftp://ftp10.us.postgresql.org/pub/postgresql/%SUBDIR%/ \
ftp://ftp13.us.postgresql.org/mirror/postgresql/%SUBDIR%/\
${MASTER_SITE_RINGSERVER:S,%SUBDIR%,misc/db/postgresql/&,}\        ftp://ftp.au.postgresql.org/pub/postgresql/%SUBDIR%/
\       ftp://ftp.at.postgresql.org/db/www.postgresql.org/pub/%SUBDIR%/ \
ftp://ftp.be.postgresql.org/postgresql/%SUBDIR%/\        ftp://ftp.ba.postgresql.org/pub/postgresql/%SUBDIR%/ \
ftp://ftp.br.postgresql.org/pub/PostgreSQL/%SUBDIR%/\        ftp://ftp3.ca.postgresql.org/pub/%SUBDIR%/ \
ftp://ftp2.cr.postgresql.org/pub/Unix/postgres/%SUBDIR%/\
ftp://ftp.cz.postgresql.org/pub/ftp.postgresql.org/%SUBDIR%/\
ftp://ftp2.cz.postgresql.org/pub/postgresql/%SUBDIR%/\        ftp://ftp.ee.postgresql.org/mirrors/postgresql/%SUBDIR%/
\       ftp://ftp.fr.postgresql.org/%SUBDIR%/ \        ftp://ftp2.fr.postgresql.org/postgresql/%SUBDIR%/ \
ftp://ftp.de.postgresql.org/mirror/postgresql/%SUBDIR%/\
ftp://ftp2.de.postgresql.org/pub/comp/os/unix/database/postgresql/%SUBDIR%/\
ftp://ftp3.de.postgresql.org/pub/Mirrors/ftp.postgresql.org/%SUBDIR%/\
ftp://ftp4.de.postgresql.org/pub/postgresql/%SUBDIR%/\
ftp://ftp.gr.postgresql.org/pub/databases/postgresql/%SUBDIR%/\        ftp://ftp.hk.postgresql.org/postgresql/%SUBDIR%/
\       ftp://ftp2.is.postgresql.org/pub/postgresql/%SUBDIR%/ \
ftp://ftp.ie.postgresql.org/mirrors/ftp.postgresql.org/pub/%SUBDIR%/\
ftp://ftp2.it.postgresql.org/mirrors/postgres/%SUBDIR%/\
ftp://ftp.kr.postgresql.org/mirror/database/postgresql/%SUBDIR%/\
ftp://ftp.lv.postgresql.org/pub/software/postgresql/%SUBDIR%/\
ftp://ftp.eu.postgresql.org/pub/unix/db/postgresql/%SUBDIR%/\
ftp://ftp2.nl.postgresql.org/mirror/postgresql/%SUBDIR%/\
ftp://ftp4.nl.postgresql.org/postgresql.zeelandnet.nl/%SUBDIR%/\
ftp://ftp.no.postgresql.org/pub/databases/postgresql/%SUBDIR%/\
ftp://ftp6.pl.postgresql.org/pub/postgresql/%SUBDIR%/\
ftp://ftp7.pl.postgresql.org/pub/mirror/ftp.postgresql.org/%SUBDIR%/\
ftp://ftp.pt.postgresql.org/postgresql/%SUBDIR%/\
ftp://ftp6.ro.postgresql.org/pub/mirrors/ftp.postgresql.org/%SUBDIR%/\
ftp://ftp.ru.postgresql.org/pub/unix/database/pgsql/%SUBDIR%/\
ftp://ftp2.ru.postgresql.org/pub/databases/postgresql/%SUBDIR%/\
ftp://ftp3.ru.postgresql.org/pub/mirror/postgresql/pub/%SUBDIR%/\
ftp://ftp5.es.postgresql.org/mirror/postgresql/%SUBDIR%/\
ftp://ftp.se.postgresql.org/pub/databases/relational/postgresql/%SUBDIR%/\
ftp://ftp2.ch.postgresql.org/pub/postgresql/%SUBDIR%/\        ftp://ftp.tw.postgresql.org/pub/postgresql/%SUBDIR%/ \
   ftp://ftp3.tw.postgresql.org/pub/postgresql/%SUBDIR%/ \
ftp://ftp.tr.postgresql.org/pub/SQL/postgresql/%SUBDIR%/\        ftp://ftp.postgresql.org/pub/%SUBDIR%/
 


----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664


Re: Some download statistics

From
"Marc G. Fournier"
Date:
On Thu, 24 Feb 2005, Tom Lane wrote:

> "Marc G. Fournier" <scrappy@postgresql.org> writes:
>> On Thu, 24 Feb 2005, Tom Lane wrote:
>>> Also: I notice that the README file that's supposed to tell people about
>>> the split-tarball scheme is not present in any of the recent-version
>>> subdirectories, so it's no wonder that they are confused.
>
>> Actually, its never  been ... just  checked, its in the root directory ...
>> but, with that in mind, I've added putting a copy in there to the release
>> script ...
>
> It is present in the subdirectories for some older releases, eg v7.3.

'k, now I see it ... I was looking for a README file .. fixed ...


----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664


Re: Some download statistics

From
"Jim C. Nasby"
Date:
On Thu, Feb 24, 2005 at 10:27:03PM +0100, Ernst Herzberg wrote:
> On Thursday 24 February 2005 21:43, Marc G. Fournier wrote:
> [.......]
> > >
> > >     pub/source/v.8.0.1/
> > >         postgresql-8.0.1.tar.bz2
> > >         postgresql-8.0.1.tar.bz2.md5
> > >         postgresql-8.0.1.tar.gz
> > >         postgresql-8.0.1.tar.gz.md5
> > >         split-tarballs/
> > >             postgresql-base-8.0.1.tar.bz2
> > >             postgresql-base-8.0.1.tar.bz2.md5
> > >             etc...
> >
> > That sounds like a reasonable compromise ... I could do that right away,
> > if nobody is in disagreement ... ?
> 
> Hm, there is a distribution that uses the split tarballs: gentoo.
> 
> snipped from /usr/portage/dev-db/postgresql/postgresql-8.0.1-r1.ebuild :
> 
> SRC_URI="mirror://postgresql/source/v${PV}/${PN}-base-${MY_PV}.tar.bz2
>     mirror://postgresql/source/v${PV}/${PN}-opt-${MY_PV}.tar.bz2
>     doc? ( mirror://postgresql/source/v${PV}/${PN}-docs-${MY_PV}.tar.bz2 )"
> 
> Better is to drop a 'bug' to bugs.gentoo.org so that the ebuild will be 
> fixed;-)

How exactly is that a bug? Why should someone spend time downloading the
docs if they don't want to install them? And gentoo isn't the only OS
that uses them.

Unless generating the splits requires a non-trivial amount of work, I
see no reason not to have them.
-- 
Jim C. Nasby, Database Consultant               decibel@decibel.org 
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"


Re: Some download statistics

From
"Marc G. Fournier"
Date:
On Fri, 25 Feb 2005, Jim C. Nasby wrote:

> Unless generating the splits requires a non-trivial amount of work, I
> see no reason not to have them.

its all automated ... still wish I could figure out a way of splitting off 
*just* libppq, since that would drop downloads of the full package 
significantly ... in my case, I need postgresql "server" on maybe one 
machine, but need just libpq on every other one that I run ..

I've gotta sit down and see if I can  figure out how to do thatt .. would 
really be interesting to see the stats if  we had a seperate 'libpq' 
download :)

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664