Thread: Web mirrors
Hi, Devrim noticed a problem earlier today with our web mirrors. The new site requires the Apache mod_rewrite module to be active on the server, and for the server to be configured to honour a .htaccess setting enabling MultiViews). If this is not configured, then some pages will give 404's on the mirrors. In considering how to handle this though, I looked at the stats recorded by the new clickthru counter used on the mirror pages. Since we went live yesterday, only 141 web mirror selections have been recorded. By contrast, there have been 6660 (!?!) clickthrus to ftp mirrors and torrent files. We currently have 58 active web mirrors, each rsyncing around 10,000 files at least daily which seems to me like a huge waste of cpu cycles and disk accesses (bandwidth should be minimal as rsync is quite efficient on the network). So, the question is, should we get rid of the bulk of the web mirrors, and leave say 5 in place around the world, which are all configured in a round-robin DNS arrangement with www.postgresql.org? These mirrors would be fast, reliable ones, hosted by people we know and trust, who are able and willing to configure the virtual host to our requirements (something we might not be able to get all the existing mirrors to do - especially if they are not running Apache, or don't have mod_rewrite compiled in). What do people think? Regards, Dave PS. I do not propose to change anything on the ftp mirror side - they are definitely worthwhile.
On Thu, 23 Dec 2004, Dave Page wrote: > Hi, > > Devrim noticed a problem earlier today with our web mirrors. The new > site requires the Apache mod_rewrite module to be active on the server, > and for the server to be configured to honour a .htaccess setting > enabling MultiViews). If this is not configured, then some pages will > give 404's on the mirrors. Is there a way of adding a test to the mirrors generating script to make sure that the mirror *is* configured correctly? Should just be a matter of modifying your current check so that if not configured right, it fails, no? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
On Thu, 23 Dec 2004, Marc G. Fournier wrote: > On Thu, 23 Dec 2004, Dave Page wrote: > >> Hi, >> >> Devrim noticed a problem earlier today with our web mirrors. The new >> site requires the Apache mod_rewrite module to be active on the server, >> and for the server to be configured to honour a .htaccess setting >> enabling MultiViews). If this is not configured, then some pages will >> give 404's on the mirrors. > > Is there a way of adding a test to the mirrors generating script to make sure > that the mirror *is* configured correctly? Should just be a matter of > modifying your current check so that if not configured right, it fails, no? > may be before any sanction better explain how to properly configure mirror ? I just tried to find out any instruction on new server but stuck with links to wwwmaster.postgresql.org, Why there are many links from www.postgresql.org to wwwmaster ? For example, on download page ftp mirror selecttion linked to http://wwwmaster.postgresql.org/download/mirrors-ftp. Also, I couldn't figure out where I could download Windows binary. > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > > ---------------------------(end of broadcast)--------------------------- > TIP 9: the planner will ignore your desire to choose an index scan if your > joining column's datatypes do not match > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
> -----Original Message----- > From: Marc G. Fournier [mailto:scrappy@postgresql.org] > Sent: 23 December 2004 18:34 > To: Dave Page > Cc: pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Web mirrors > > Is there a way of adding a test to the mirrors generating > script to make > sure that the mirror *is* configured correctly? Should just > be a matter > of modifying your current check so that if not configured > right, it fails, > no? Yeah, I can do that relatively easily (probably not before new year though), but it seems a waste of resources to be syncing 58 mirrors that are barely used - hence my suggestion. Can any of the mirror operators give an idea of the traffic they receive? To www.uk.postgresql.org it's neligible, though I haven't counted anything yet. Regards, Dave.
> -----Original Message-----
> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
> Sent: 23 December 2004 19:51
> To: Marc G. Fournier
> Cc: Dave Page; pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] Web mirrors
>
> may be before any sanction better explain how to properly configure
> mirror ? I just tried to find out any instruction on new server but
I *think* you need something like:
<Directory /path/to/docroot>
AllowOverride All
</Directory>
And mod_rewrite needs to be enabled. If someone can confirm that is the
case ASAP, I will email all the mirror owners tomorrow and ask them to
make sure that is in their config so at least as many as possible work
for the time being.
> stuck with links to wwwmaster.postgresql.org, Why there are many links
> from www.postgresql.org to wwwmaster ? For example,
> on download page ftp mirror selecttion linked to
> http://wwwmaster.postgresql.org/download/mirrors-ftp.
Because the ip2country stuff is on wwwmaster, as is the clickthru
tracker that records the usage of the mirrors and other urls that may be
clicked. That's how I can tell how many people have selected a web
mirror. Without running php code on each mirror, and granting them
access to the database, that code must be centralised. As I have sai in
the past when it was discussed, at the first sign of any performance
problems etc. I will revert it to a static system, without ip2country,
stats logging.
> Also, I couldn't figure out where I could download Windows binary.
One of the ftp mirrors, just like any other binary release. So far as I
can tell from past discussions most people think that the Windows
binaries should not be treated any differently from any others. It is a
close thing though, and at the moment there is a symlink to the
directory from the top of the ftp tree.
Regards, Dave.
On Thu, 23 Dec 2004, Dave Page wrote: > > >> -----Original Message----- >> From: Marc G. Fournier [mailto:scrappy@postgresql.org] >> Sent: 23 December 2004 18:34 >> To: Dave Page >> Cc: pgsql-www@postgresql.org >> Subject: Re: [pgsql-www] Web mirrors >> >> Is there a way of adding a test to the mirrors generating >> script to make >> sure that the mirror *is* configured correctly? Should just >> be a matter >> of modifying your current check so that if not configured >> right, it fails, >> no? > > Yeah, I can do that relatively easily (probably not before new year > though), but it seems a waste of resources to be syncing 58 mirrors that > are barely used - hence my suggestion. Except you are assuming that beyond that first connection, they aren't bookmarking the mirror itself, and nevr hitting the mirrors page to click thru ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
> -----Original Message----- > From: Marc G. Fournier [mailto:scrappy@postgresql.org] > Sent: 24 December 2004 02:33 > To: Dave Page > Cc: Marc G. Fournier; pgsql-www@postgresql.org > Subject: RE: [pgsql-www] Web mirrors > > Except you are assuming that beyond that first connection, > they aren't > bookmarking the mirror itself, and nevr hitting the mirrors > page to click > thru ... Yes, I realise that - that's why I asked for some stats :-) /D
Hi,
Dave Page wrote:
>>may be before any sanction better explain how to properly configure
>>mirror ? I just tried to find out any instruction on new server but
>
>
> I *think* you need something like:
>
> <Directory /path/to/docroot>
> AllowOverride All
> </Directory>
I think this is a bit too much,
<Directory /path/to/docroot>
AllowOverride Options
</Directory>
will be enough if we provide our own .htaccess, and
<Directory /path/to/docroot>
Options MultiViews
</Directory>
will be sufficient if we do not.
.htaccess should look like this:
Options MultiViews
AddDefaultCharset utf-8
> And mod_rewrite needs to be enabled. If someone can confirm that is the
> case ASAP, I will email all the mirror owners tomorrow and ask them to
> make sure that is in their config so at least as many as possible work
> for the time being.
IIRC we use mod_rewrite only on dynamic website. It is not needed on static mirrors.
> -----Original Message----- > From: Alexey Borzov [mailto:borz_off@cs.msu.su] > Sent: 24 December 2004 09:06 > To: Dave Page > Cc: Oleg Bartunov; Marc G. Fournier; pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Web mirrors > > Hi, > > Dave Page wrote: > >>may be before any sanction better explain how to properly configure > >>mirror ? I just tried to find out any instruction on new server but > > > > > > I *think* you need something like: > > > > <Directory /path/to/docroot> > > AllowOverride All > > </Directory> > > I think this is a bit too much, > > <Directory /path/to/docroot> > AllowOverride Options > </Directory> > will be enough if we provide our own .htaccess, and OK, (and yes we do provide a .htaccess) > .htaccess should look like this: > Options MultiViews > AddDefaultCharset utf-8 We don't include the latter. I'll fix that. > > And mod_rewrite needs to be enabled. If someone can confirm > that is the > > case ASAP, I will email all the mirror owners tomorrow and > ask them to > > make sure that is in their config so at least as many as > possible work > > for the time being. > > IIRC we use mod_rewrite only on dynamic website. It is not > needed on static mirrors. Oh, OK. I assumed it was used to do the multiview rewriting (ie index.html -> index.html.xx). Thanks Alexey. Regards, Dave.