On Mon, 5 Jan 2004, Marek Lewczuk wrote:
> Dave Cramer wrote:
> > connection failed :(
> works for me... :-) (poland)
>
We have small downtime because of upgrading server software, so this may
be a reason for the problem. We're in stage of optimizing crawler because
some sites are very-very ugly, for example, our crawler have discovered
2 millions URLs on http://ems-hitech.com/pgmanager/ ! 99.99 % of URLs are
just 404 (document not found), but server does return 200 code )\:)
So we have to explicitly exclude these pages. btw, archives.postgresql.org
doesn't returns modification date in header. This prevent crawler to
optimize downloading process. So, there are many problems, but we hope
soon we'll tune crawling process. I estimate average time to refresh index
about 1 week.
>
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
