Thread: Inadequate hosting for www.postgresql.org?
Hi, I'm testing the mirroring script for new version of postgresql.org and the results are *interesting*: Oct 31 10:11:32 mirror [info] Mirroring finished. 423 page(s) saved, 1226 second(s) spent ~3 seconds per page is a bit too much, don't you think? Also please note that it took ~20 minutes to save the pages *not including* the documentation, which is *several thousand* pages in size! Here are the results for mirroring of just 7.4 "interactive" docs: Oct 31 11:12:46 mirror [info] Mirroring finished. 603 page(s) saved, 2716 second(s) spent That's >4 seconds per page... These are the pages from wwwdevel, but as www and wwwdevel are on one box physically, you get the unpleasant picture. It actually worked faster on my test server and that is P2-233. Also please note that today is Sunday, I doubt it will be faster on weekday. Dunno about you, but if that was my site I'd already be searching for another hosting. I'm sure that some hosters will be glad to even donate that to PostgreSQL project.
> > Dunno about you, but if that was my site I'd already be searching for > another hosting. I'm sure that some hosters will be glad to even > donate that to PostgreSQL project. This has been brought multiple times in the past. Command Prompt and Gavin both have offered hosting resources. Command Prompt already hosts the archives. I know Gavin does something but I can't recall what (sorry Gavin ;). The resources are available but it is up to the project to request them. Sincerely, Joshua D. Drake > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org -- Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC Postgresql support, programming shared hosting and dedicated hosting. +1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com PostgreSQL Replicator -- production quality replication for PostgreSQL
Attachment
That's why mirrors are pulled from http://www.postgresql.org, nor are they pulled using what I'd guess you are using ( wget? ) below, but using rsync ... hasn't been for years now ... In fact, over the next little while, http://www.postgresql.org will actually be pointing to an RR DNS entry, and not even the 'base server' ... I've just gotta sit down and update the DNS generation scripts for this ... On Sun, 31 Oct 2004, Alexey Borzov wrote: > Hi, > > I'm testing the mirroring script for new version of postgresql.org and the > results are *interesting*: > > Oct 31 10:11:32 mirror [info] Mirroring finished. 423 page(s) saved, 1226 > second(s) spent > > ~3 seconds per page is a bit too much, don't you think? Also please note that > it took ~20 minutes to save the pages *not including* the documentation, > which is *several thousand* pages in size! > > Here are the results for mirroring of just 7.4 "interactive" docs: > > Oct 31 11:12:46 mirror [info] Mirroring finished. 603 page(s) saved, 2716 > second(s) spent > > That's >4 seconds per page... > > These are the pages from wwwdevel, but as www and wwwdevel are on one box > physically, you get the unpleasant picture. It actually worked faster on my > test server and that is P2-233. Also please note that today is Sunday, I > doubt it will be faster on weekday. > > Dunno about you, but if that was my site I'd already be searching for another > hosting. I'm sure that some hosters will be glad to even donate that to > PostgreSQL project. > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: > That's why mirrors are pulled from http://www.postgresql.org, nor are > they pulled using what I'd guess you are using ( wget? ) below, but > using rsync ... hasn't been for years now ... > > In fact, over the next little while, http://www.postgresql.org will > actually be pointing to an RR DNS entry, and not even the 'base server' > ... I've just gotta sit down and update the DNS generation scripts for > this ... Uh, seems I was unclear about what I was doing. I understand of course that the site is mirrored via rsync, but consider the following: the site which resides in CVS is just a few PHP scripts, to have something to rsync you first need to *generate* the static pages. And here comes the problem: page generation is outrageously slow. In current version special scripts are used to generate static pages, but these are terrible hacks, f.e. you have to make the same change to design both in the dynamic page and the mirroring script. But they maybe run in acceptable time, I don't know, ask Dave. For the next version I just wrote a generic mirroring script behaving like wget but taking into account some specific needs (like i18n). The plus is that you don't need to change this script *at all* if you add pages or even completely redesign the site. But now it looks as if generation of full static website will take several hours. And no, this has nothing to do with network latency, the mirror generation script was run locally. Marc, page generation times of 3+ seconds are only acceptable if you do some extremely complex database queries in them, think reports. This is definitely *not* the case with the code of postgresql.org, my experience tells me that modern hardware can happily churn 5-10 of such pages *per second*. Thus: either hardware is completely inadequate or the servers are overloaded.
I just email'd Alexey to test after making some changes (moving the whole VM to a faster server) and reporting much better response times ... On Sun, 31 Oct 2004, Alexey Borzov wrote: > Hi, > > Marc G. Fournier wrote: >> That's why mirrors are pulled from http://www.postgresql.org, nor are they >> pulled using what I'd guess you are using ( wget? ) below, but using rsync >> ... hasn't been for years now ... >> >> In fact, over the next little while, http://www.postgresql.org will >> actually be pointing to an RR DNS entry, and not even the 'base server' ... >> I've just gotta sit down and update the DNS generation scripts for this ... > > Uh, seems I was unclear about what I was doing. > > I understand of course that the site is mirrored via rsync, but consider the > following: the site which resides in CVS is just a few PHP scripts, to have > something to rsync you first need to *generate* the static pages. And here > comes the problem: page generation is outrageously slow. > > In current version special scripts are used to generate static pages, but > these are terrible hacks, f.e. you have to make the same change to design > both in the dynamic page and the mirroring script. But they maybe run in > acceptable time, I don't know, ask Dave. > > For the next version I just wrote a generic mirroring script behaving like > wget but taking into account some specific needs (like i18n). The plus is > that you don't need to change this script *at all* if you add pages or even > completely redesign the site. > > But now it looks as if generation of full static website will take several > hours. And no, this has nothing to do with network latency, the mirror > generation script was run locally. > > Marc, page generation times of 3+ seconds are only acceptable if you do some > extremely complex database queries in them, think reports. This is definitely > *not* the case with the code of postgresql.org, my experience tells me that > modern hardware can happily churn 5-10 of such pages *per second*. Thus: > either hardware is completely inadequate or the servers are overloaded. > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Marc, I think main problem is available memory, not cpu. Do you have single-server setup for www.postgresql.org ? I see from headers zen:~/downloads$>curl -I http://www.postgresql.org/ HTTP/1.1 200 OK Date: Sun, 31 Oct 2004 21:13:50 GMT Server: Apache/1.3.31 (Unix) PHP/4.3.9 Last-Modified: Sun, 31 Oct 2004 21:00:07 GMT Do you have persistent connection from php to database ? What about cacheing of dynamic pages ? Also, I recommend webperf.org to see how fast selected page is loaded http://www.webperf.org/breakdown.html?URL=www.postgresql.org Connect DNS FirstByte End Size(Bytes) URL Status 0.033 0.071 0.383 1.073 21429 http://www.postgresql.org 200 0.412 0.412 1.161 1.270 4447 http://www.postgresql.org/images/banner-left.jpg 200 0.532 0.532 1.152 1.254 2929 http://www.postgresql.org/images/banner-right.jpg 200 0.650 0.650 1.143 1.146 929 http://www.postgresql.org/postgresql.css 200 0.768 0.793 1.207 1.418 8616 http://ads.postgresql.org/display_image.php?lvl=sponsor 200 0.905 0.905 1.156 1.277 7465 http://ads.postgresql.org/display_image.php?lvl=free 200 Oleg On Sun, 31 Oct 2004, Marc G. Fournier wrote: > > I just email'd Alexey to test after making some changes (moving the whole VM > to a faster server) and reporting much better response times ... > > On Sun, 31 Oct 2004, Alexey Borzov wrote: > >> Hi, >> >> Marc G. Fournier wrote: >>> That's why mirrors are pulled from http://www.postgresql.org, nor are they >>> pulled using what I'd guess you are using ( wget? ) below, but using rsync >>> ... hasn't been for years now ... >>> >>> In fact, over the next little while, http://www.postgresql.org will >>> actually be pointing to an RR DNS entry, and not even the 'base server' >>> ... I've just gotta sit down and update the DNS generation scripts for >>> this ... >> >> Uh, seems I was unclear about what I was doing. >> >> I understand of course that the site is mirrored via rsync, but consider >> the following: the site which resides in CVS is just a few PHP scripts, to >> have something to rsync you first need to *generate* the static pages. And >> here comes the problem: page generation is outrageously slow. >> >> In current version special scripts are used to generate static pages, but >> these are terrible hacks, f.e. you have to make the same change to design >> both in the dynamic page and the mirroring script. But they maybe run in >> acceptable time, I don't know, ask Dave. >> >> For the next version I just wrote a generic mirroring script behaving like >> wget but taking into account some specific needs (like i18n). The plus is >> that you don't need to change this script *at all* if you add pages or even >> completely redesign the site. >> >> But now it looks as if generation of full static website will take several >> hours. And no, this has nothing to do with network latency, the mirror >> generation script was run locally. >> >> Marc, page generation times of 3+ seconds are only acceptable if you do >> some extremely complex database queries in them, think reports. This is >> definitely *not* the case with the code of postgresql.org, my experience >> tells me that modern hardware can happily churn 5-10 of such pages *per >> second*. Thus: either hardware is completely inadequate or the servers are >> overloaded. >> > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > > ---------------------------(end of broadcast)--------------------------- > TIP 8: explain analyze is your friend > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
Hi, Marc G. Fournier wrote: > I just email'd Alexey to test after making some changes (moving the > whole VM to a faster server) and reporting much better response times ... Yes, now it does ~5 pages per second, thus the whole mirroring process will probably complete in under an hour.
On Mon, 1 Nov 2004, Alexey Borzov wrote: > Hi, > > Marc G. Fournier wrote: >> I just email'd Alexey to test after making some changes (moving the whole >> VM to a faster server) and reporting much better response times ... > > Yes, now it does ~5 pages per second, thus the whole mirroring process will > probably complete in under an hour. Does the mirroring process mirror *all* files, or just those that have changed? Most of the files on the site *should* be static, no? What consititutes the bulk of the pages? I would think the various versions docs, no? Which are static ... BTW, does this script you are working on have any access controls, or does it allow *everyone* to mirror the site? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: >> Yes, now it does ~5 pages per second, thus the whole mirroring process >> will probably complete in under an hour. > > Does the mirroring process mirror *all* files, or just those that have > changed? Most of the files on the site *should* be static, no? What > consititutes the bulk of the pages? I would think the various versions > docs, no? Which are static ... All files except docs, since there are dynamic parts like survey and particularly gborg stats on all pages. But there are not very many pages here. As for the docs, there is version without user comments which will not need to be mirrored unless the design changes or the docs for the newer version replace the older docs. The version with user comments will need to be mirrored every time, obviously. Though we may consider removing comment form from docs for the older versions and ease this requirement. Of course one can add Last-Modified header and 304 response to the pages, but there still will be a request to make. > BTW, does this script you are working on have any access controls, or > does it allow *everyone* to mirror the site? If you mean whether anyone will be able to access the script via web then the answer is "no", it will be run from console. If the question is different then I don't get it.
On Mon, 1 Nov 2004, Alexey Borzov wrote: >> BTW, does this script you are working on have any access controls, or does >> it allow *everyone* to mirror the site? > > If you mean whether anyone will be able to access the script via web > then the answer is "no", it will be run from console. If the question is > different then I don't get it. 'k, then I'm confused ... what exactly does this script do? or, rather, what is it replacing? building the static files for mirroring purposes? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: >>> BTW, does this script you are working on have any access controls, or >>> does it allow *everyone* to mirror the site? >> >> If you mean whether anyone will be able to access the script via web >> then the answer is "no", it will be run from console. If the question >> is different then I don't get it. > > 'k, then I'm confused ... what exactly does this script do? or, rather, > what is it replacing? building the static files for mirroring purposes? Exactly. Sorry for confusing you. :)
Hi, Alexey Borzov wrote: >> I just email'd Alexey to test after making some changes (moving the >> whole VM to a faster server) and reporting much better response times ... > > > Yes, now it does ~5 pages per second, thus the whole mirroring process > will probably complete in under an hour. Hmm, looks like I was too fast to celebrate: Nov 01 11:27:51 mirror [info] Mirroring finished. 423 page(s) saved, 2407 second(s) spent Something is still very wrong with wwwdevel
Am looking into this with Alexey offlist ... On Mon, 1 Nov 2004, Alexey Borzov wrote: > Hi, > > Alexey Borzov wrote: >>> I just email'd Alexey to test after making some changes (moving the whole >>> VM to a faster server) and reporting much better response times ... >> >> >> Yes, now it does ~5 pages per second, thus the whole mirroring process will >> probably complete in under an hour. > > Hmm, looks like I was too fast to celebrate: > > Nov 01 11:27:51 mirror [info] Mirroring finished. 423 page(s) saved, 2407 > second(s) spent > > Something is still very wrong with wwwdevel > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: > Am looking into this with Alexey offlist ... Looking into this stuff offlist highlighted the following obvious problem: postgresql.org (as well as wwwdevel) runs on a shared server that has *lots* of other tasks that can negatively affect performance. It is difficult to even point finger at one particular task that does this. So the solution (as was also pointed out in a private mail to me) is to get a dedicated server for running the "master" website. I am not in the position to request such a donation, but PostgreSQL core and/or community is. > On Mon, 1 Nov 2004, Alexey Borzov wrote: >> Hmm, looks like I was too fast to celebrate: >> >> Nov 01 11:27:51 mirror [info] Mirroring finished. 423 page(s) saved, >> 2407 second(s) spent >> >> Something is still very wrong with wwwdevel
> -----Original Message----- > From: pgsql-www-owner@postgresql.org > [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Alexey Borzov > Sent: 02 November 2004 10:25 > To: Marc G. Fournier > Cc: pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Inadequate hosting for www.postgresql.org? > > Hi, > > Marc G. Fournier wrote: > > Am looking into this with Alexey offlist ... > > Looking into this stuff offlist highlighted the following obvious > problem: postgresql.org (as well as wwwdevel) runs on a > shared server that has *lots* of other tasks that can > negatively affect performance. > It is difficult to even point finger at one particular task > that does this. > > So the solution (as was also pointed out in a private mail to > me) is to get a dedicated server for running the "master" website. A /dedicated/ server is *way* over the top as I pointed out in my previous private message to you. The current site regens all 7000+ pages in a few minutes for example. I agree though that we could use a little more horsepower though - I'll see if I can setup something on one of my boxes here over the next few days. Regards, Dave
>A /dedicated/ server is *way* over the top as I pointed out in my >previous private message to you. The current site regens all 7000+ pages >in a few minutes for example. I agree though that we could use a little >more horsepower though - I'll see if I can setup something on one of my >boxes here over the next few days. > > Hello, A /dedicated/ server may be more than what WWW needs, but it is not more than it deserves. Although I do not know what else is going on with the WWW machine, a dedicated machine definitely would not hurt. Sincerely, Joshua D. Drake >Regards, Dave > >---------------------------(end of broadcast)--------------------------- >TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > > -- Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC Postgresql support, programming shared hosting and dedicated hosting. +1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com PostgreSQL Replicator -- production quality replication for PostgreSQL
Attachment
Hi, Dave Page wrote: >>Looking into this stuff offlist highlighted the following obvious >>problem: postgresql.org (as well as wwwdevel) runs on a >>shared server that has *lots* of other tasks that can >>negatively affect performance. >>It is difficult to even point finger at one particular task >>that does this. >> >>So the solution (as was also pointed out in a private mail to >>me) is to get a dedicated server for running the "master" website. > > > A /dedicated/ server is *way* over the top as I pointed out in my > previous private message to you. The current site regens all 7000+ pages > in a few minutes for example. I agree though that we could use a little > more horsepower though - I'll see if I can setup something on one of my > boxes here over the next few days. "Dedicated" does not necessarily mean "dedicated to www only", but "dedicated to PostgreSQL project". Current infrastructure with shared servers was already found inadequate to run search, archives, etc.
On Tue, 2 Nov 2004, Alexey Borzov wrote: > Hi, > > Dave Page wrote: >>> Looking into this stuff offlist highlighted the following obvious >>> problem: postgresql.org (as well as wwwdevel) runs on a shared server that >>> has *lots* of other tasks that can negatively affect performance. It is >>> difficult to even point finger at one particular task that does this. >>> >>> So the solution (as was also pointed out in a private mail to me) is to >>> get a dedicated server for running the "master" website. >> >> >> A /dedicated/ server is *way* over the top as I pointed out in my >> previous private message to you. The current site regens all 7000+ pages >> in a few minutes for example. I agree though that we could use a little >> more horsepower though - I'll see if I can setup something on one of my >> boxes here over the next few days. > > "Dedicated" does not necessarily mean "dedicated to www only", but "dedicated > to PostgreSQL project". Current infrastructure with shared servers was > already found inadequate to run search, archives, etc. You still haven't answered any of the questions that I asked in a private email ... Is this a time of day issue (from your two samples, it looks like it is)? If you set it up to run hourly, what do the #s look like for each run throughout the day? What does the loadavg look like on the server when you are running the script? Or is the database being slow? Considering that Dave states above that the current script takes minutes to generate >7000 pages, what are you doing differently that makes it so much slower? And, why exactly are we changing from the current method if the new method is going to require a dedicated server to run it? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: > You still haven't answered any of the questions that I asked in a > private email ... Is this a time of day issue (from your two samples, it > looks like it is)? If you set it up to run hourly, what do the #s look > like for each run throughout the day? What does the loadavg look like > on the server when you are running the script? Or is the database being > slow? I tried running mirror later today: Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354 second(s) spent I also am running it now, getting the same ~5 sec per page response time and uptime command states: 7:49PM up 10 days, 4:36, 4 users, load averages: 4.00, 3.22, 3.04 The database itself does not look slow, nothing like these 5 seconds to connect / send the standard query. > Considering that Dave states above that the current script takes minutes > to generate >7000 pages, what are you doing differently that makes it so > much slower? And, why exactly are we changing from the current method > if the new method is going to require a dedicated server to run it? Yes, that's one way to ask these questions. The other way is: why are we having response times of ~5 seconds for not-too-complex pages? The only thing I'm doing differently from the current one is that I'm sending HTTP requests to get the pages. I already outlined benefits of this approach either here or in a private mail.
On Tue, 2 Nov 2004, Alexey Borzov wrote: > Hi, > > Marc G. Fournier wrote: >> You still haven't answered any of the questions that I asked in a private >> email ... Is this a time of day issue (from your two samples, it looks like >> it is)? If you set it up to run hourly, what do the #s look like for each >> run throughout the day? What does the loadavg look like on the server when >> you are running the script? Or is the database being slow? > > I tried running mirror later today: > > Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354 > second(s) spent > > I also am running it now, getting the same ~5 sec per page response time and > uptime command states: > > 7:49PM up 10 days, 4:36, 4 users, load averages: 4.00, 3.22, 3.04 > > The database itself does not look slow, nothing like these 5 seconds to > connect / send the standard query. > >> Considering that Dave states above that the current script takes minutes to >> generate >7000 pages, what are you doing differently that makes it so much >> slower? And, why exactly are we changing from the current method if the >> new method is going to require a dedicated server to run it? > > Yes, that's one way to ask these questions. The other way is: why are we > having response times of ~5 seconds for not-too-complex pages? Your guess is as good as mine ... using my browser from here, I'm not seen 5sec delays, and I'm two countries away from it ... using lynx on the VM itself, goign through the web site, I'm seeing instantaneous responses ... I'm definitely not seeing 5sec delays ... Can you extend your script so that it outputs URL spider'd and time taken in msec for it? The pages that I'm testing definitely aren't giving me 5sec lag times from the server itself ... so if you can give me a 'slow URL' that I can look at, that would help ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
On Tue, 2 Nov 2004, Marc G. Fournier wrote: > On Tue, 2 Nov 2004, Alexey Borzov wrote: > >> Hi, >> >> Marc G. Fournier wrote: >>> You still haven't answered any of the questions that I asked in a private >>> email ... Is this a time of day issue (from your two samples, it looks >>> like it is)? If you set it up to run hourly, what do the #s look like for >>> each run throughout the day? What does the loadavg look like on the >>> server when you are running the script? Or is the database being slow? >> >> I tried running mirror later today: >> >> Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354 >> second(s) spent >> >> I also am running it now, getting the same ~5 sec per page response time >> and uptime command states: >> >> 7:49PM up 10 days, 4:36, 4 users, load averages: 4.00, 3.22, 3.04 >> >> The database itself does not look slow, nothing like these 5 seconds to >> connect / send the standard query. >> >>> Considering that Dave states above that the current script takes minutes >>> to generate >7000 pages, what are you doing differently that makes it so >>> much slower? And, why exactly are we changing from the current method if >>> the new method is going to require a dedicated server to run it? >> >> Yes, that's one way to ask these questions. The other way is: why are we >> having response times of ~5 seconds for not-too-complex pages? > > Your guess is as good as mine ... using my browser from here, I'm not seen > 5sec delays, and I'm two countries away from it ... using lynx on the VM > itself, goign through the web site, I'm seeing instantaneous responses ... > I'm definitely not seeing 5sec delays ... > > Can you extend your script so that it outputs URL spider'd and time taken in > msec for it? The pages that I'm testing definitely aren't giving me 5sec lag > times from the server itself ... so if you can give me a 'slow URL' that I > can look at, that would help ... 'k, I just looked at your mirror.php script .. and realized that its not the main web site thats the problem, its your new site ... why is your site so much slower to load up with lynx then the site that is there now? http://www.postgresql.org comes up instantly, http://www.alexey.beta.postgresql.org/index.html.en takes forever to load ... why? I can actually get coffee waiting for your site to load in my browser here, where there is almost zero delay loading the current site ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
> -----Original Message----- > From: Marc G. Fournier [mailto:scrappy@postgresql.org] > Sent: 02 November 2004 20:49 > To: Marc G. Fournier > Cc: Alexey Borzov; Dave Page; pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Inadequate hosting for www.postgresql.org? > > 'k, I just looked at your mirror.php script .. and realized > that its not the main web site thats the problem, its your > new site ... why is your site so much slower to load up with > lynx then the site that is there now? > http://www.postgresql.org comes up instantly, > http://www.alexey.beta.postgresql.org/index.html.en takes > forever to load ... why? You should be comparing with wwwdevel.postgresql.org, not www.postgresql.org really. But still, it's a good question... /D
Hi, Marc G. Fournier wrote: > 'k, I just looked at your mirror.php script .. and realized that its not > the main web site thats the problem, its your new site ... why is your > site so much slower to load up with lynx then the site that is there > now? http://www.postgresql.org comes up instantly, > http://www.alexey.beta.postgresql.org/index.html.en takes forever to > load ... why? That's easy: www.postgresql.org is static HTML, alexey.beta is dynamic. I added profiling information to the pages (look at the bottom, only the index page has a full set of "markers") and the results are the following: ~80% of the page generation time (which is 3+ seconds!) is spent loading local files and parsing and executing PHP code. Database access is actually *very fast*. DB connection is slow, but that takes only ~10% of time and can be fixed by using persistant connections. That also answers the question why current generation scripts work in acceptable time. BTW, may there be some problems with local file access on this setup? Dave may test the same on wwwdevel if he copies system/handler.php system/page.php system/page/news-and-events.php from alexey.beta's webroot to wwwdevel, though I doubt the results will have any significant difference... > I can actually get coffee waiting for your site to load in my browser > here, where there is almost zero delay loading the current site ... OK, here is the answer: the box alexey.beta is upon is only capable of serving static content. It chokes on even moderately dynamic one. Buy some real hardware, it's not that pricey now.
Its interesting to see how webperf.org sees different postgresql.org pages. For example, http://www.webperf.org/breakdown.html?URL=www.alexey.beta.postgresql.org Connect DNS FirstByte End Size(Bytes) URL Status 0.033 0.074 3.226 4.225 17654 http://www.alexey.beta.postgresql.org 200 3.258 3.258 3.854 3.933 43 http://www.alexey.beta.postgresql.org/layout/images/0.gif 200 3.379 3.379 3.850 4.356 8190 http://www.alexey.beta.postgresql.org/layout/images/logo.png 200 3.519 3.520 3.930 4.070 2101 http://www.alexey.beta.postgresql.org/layout/css/new.css 200 3.710 3.729 4.067 4.363 11470 http://ads.postgresql.org/display_image.php?lvl=sponsor 200 3.935 3.936 8.678 8.801 2644 http://www.alexey.beta.postgresql.org/news.rss 200 4.072 4.072 7.736 7.739 864 http://www.alexey.beta.postgresql.org/events.rss 200 4.228 4.228 4.496 4.499 207 http://www.alexey.beta.postgresql.org/layout/images/mark_menu.png 200 It shows, that possible problem could be news.rss, events.rss. www.postgresql.org behaves much better http://www.webperf.org/breakdown.html?URL=http%3A%2F%2Fwww.postgresql.org Connect DNS FirstByte End Size(Bytes) URL Status 0.033 0.074 0.403 1.118 21429 http://www.postgresql.org 200 0.417 0.417 1.082 1.099 929 http://www.postgresql.org/postgresql.css 200 0.581 0.581 1.220 1.354 4447 http://www.postgresql.org/images/banner-left.jpg 200 0.700 0.700 1.211 1.336 2929 http://www.postgresql.org/images/banner-right.jpg 200 0.823 0.841 1.259 1.522 14397 http://ads.postgresql.org/display_image.php?lvl=sponsor 200 0.955 0.955 1.252 1.498 14397 http://ads.postgresql.org/display_image.php?lvl=free 200 But, I again want to stress that current architecture of postgresql.org sites are very-very bad and you will never get nice performance and scalability especially for dynamic sites without: 1. Redirecting different traffic to different servers. a)images and other binary things should be served by small and fast server like thttpd or lighttpd. Currently, images are served by apache+php server which is wasting of resources ! These servers could easily serve several thousands req/sec, so one images.postgresql.org could serve all such kind of traffic. b)requests for dynamic pages should be forwarded to backend servers with php compiled and keeping persistent connection to db. These backends are fully hidden and accessable only by localhost, so they will not stay busy while serving could-be-slow client (slow connectivity), That means resources (memory) will be not wasted and there is no demand to have many servers online, because interaction with clients will be server only by frontends, see below c)requests for static pages are served by plain httpd server like apache - frontend. These servers should have cacheing capability to cache answers from dynamic servers, so many-many requests for dynamic pages will not actualy forwarded to heavy backends ! Our pages are not actually very dynamic. This is a classical 3 servers setup and it described in many books and success stories. Oleg On Tue, 2 Nov 2004, Marc G. Fournier wrote: > On Tue, 2 Nov 2004, Marc G. Fournier wrote: > >> On Tue, 2 Nov 2004, Alexey Borzov wrote: >> >>> Hi, >>> >>> Marc G. Fournier wrote: >>>> You still haven't answered any of the questions that I asked in a private >>>> email ... Is this a time of day issue (from your two samples, it looks >>>> like it is)? If you set it up to run hourly, what do the #s look like >>>> for each run throughout the day? What does the loadavg look like on the >>>> server when you are running the script? Or is the database being slow? >>> >>> I tried running mirror later today: >>> >>> Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354 >>> second(s) spent >>> >>> I also am running it now, getting the same ~5 sec per page response time >>> and uptime command states: >>> >>> 7:49PM up 10 days, 4:36, 4 users, load averages: 4.00, 3.22, 3.04 >>> >>> The database itself does not look slow, nothing like these 5 seconds to >>> connect / send the standard query. >>> >>>> Considering that Dave states above that the current script takes minutes >>>> to generate >7000 pages, what are you doing differently that makes it so >>>> much slower? And, why exactly are we changing from the current method if >>>> the new method is going to require a dedicated server to run it? >>> >>> Yes, that's one way to ask these questions. The other way is: why are we >>> having response times of ~5 seconds for not-too-complex pages? >> >> Your guess is as good as mine ... using my browser from here, I'm not seen >> 5sec delays, and I'm two countries away from it ... using lynx on the VM >> itself, goign through the web site, I'm seeing instantaneous responses ... >> I'm definitely not seeing 5sec delays ... >> >> Can you extend your script so that it outputs URL spider'd and time taken >> in msec for it? The pages that I'm testing definitely aren't giving me >> 5sec lag times from the server itself ... so if you can give me a 'slow >> URL' that I can look at, that would help ... > > 'k, I just looked at your mirror.php script .. and realized that its not the > main web site thats the problem, its your new site ... why is your site so > much slower to load up with lynx then the site that is there now? > http://www.postgresql.org comes up instantly, > http://www.alexey.beta.postgresql.org/index.html.en takes forever to load ... > why? > > I can actually get coffee waiting for your site to load in my browser here, > where there is almost zero delay loading the current site ... > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
Hi, Oleg Bartunov wrote: > It shows, that possible problem could be news.rss, events.rss. That's because these pages are dynamic, too. But I think it's an error in webperf that it considers them part of index page. > This is a classical 3 servers setup and it described in many books and > success stories. Your suggestion is nice and all, but right now we are discussing the fact that we don't even have *one* dedicated server. Only a shared account on substandard hardware, incapable of running PHP scripts. :(
On Wed, 3 Nov 2004, Alexey Borzov wrote: > Hi, > > Oleg Bartunov wrote: >> It shows, that possible problem could be news.rss, events.rss. > > That's because these pages are dynamic, too. But I think it's an error in > webperf that it considers them part of index page. > >> This is a classical 3 servers setup and it described in many books and >> success stories. > > Your suggestion is nice and all, but right now we are discussing the fact > that we don't even have *one* dedicated server. Only a shared account on > substandard hardware, incapable of running PHP scripts. :( This has yet to be proven, IMHO ... Dave has already stated that the old build scripts aren't nearly as slow as yours, and it was having to deal with PHP as well ... Your first "complaint" was that it wasn't loading the files fast enough ... so, I enabled mmcache on that server with shared memory use only, so that it doesn't use disk except for the first load ... the build is still atrociously slow ... the servers are all setup so that shared memory is locked into physical RAM, so now what? The RAM is too slow? If anyone wants to look at the PHP settings on that server, check out http://www.postgresql.org/phpinfo.php ... is there something that should be different on compile? or something that is turned on that might cause a slowdown? I'm using pretty much default FreeBSD ports options, in so far as build and php.ini is concerned ... maybe there is something in there that should be changed to optimize better? The server itself is doing neglible swapping (<6M) and unless I've been reading top wrong all my life (which is definitely a possibility) its reporting 1G of 'Inact' RAM, which I believe means that available for processes ... ? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Hi, Marc G. Fournier wrote: >> That's because these pages are dynamic, too. But I think it's an error >> in webperf that it considers them part of index page. >> >>> This is a classical 3 servers setup and it described in many >>> books and >>> success stories. >> >> >> Your suggestion is nice and all, but right now we are discussing the >> fact that we don't even have *one* dedicated server. Only a shared >> account on substandard hardware, incapable of running PHP scripts. :( > > This has yet to be proven, IMHO ... Dave has already stated that the old > build scripts aren't nearly as slow as yours, and it was having to deal > with PHP as well ... > > Your first "complaint" was that it wasn't loading the files fast enough > ... so, I enabled mmcache on that server with shared memory use only, so > that it doesn't use disk except for the first load ... the build is > still atrociously slow ... the servers are all setup so that shared > memory is locked into physical RAM, so now what? The RAM is too slow? In case I was unclear before: due to some f**kup in server configuration (probably well below the PHP level) file access is EXTREMELY slow. And there *is* file access now due to templates and gettext, even if PHP scripts are cached. If you look at profiling information at http://alexey.beta.postgresql.org/, you'll see that just reading files (loadedTemplate, loadedInnerTemplate, loadedIndexTemplate) takes 0.2--0.5 seconds. Initializing gettext (which relies on files, too) takes 0.1 seconds (setLanguage). One also has to wonder whether there is another configuration f**kup that makes script connect to PostgreSQL for 0.1--0.5 seconds. One *also* has to wonder what (if not server overload) makes these times fluctuate a lot. A well tuned webserver on adequate hardware will generate the whole page of alexey.beta in less than 0.1 second. Current build scripts DO NOT DO FILE ACCESS, that's why they work faster. But that's irrelevant, 'cause there are pages on new website that should be dynamic and thus will be accessed by visitors on the master server. Page generation times of ~2 seconds will make PostgreSQL project look BAD. As for the nature of aforementioned configuration f**kups --- no idea. *You* are the admin.
On Wed, 03 Nov 2004 13:35:56 +0300, Alexey Borzov <borz_off@cs.msu.su> wrote: > Your suggestion is nice and all, but right now we are discussing the fact that > we don't even have *one* dedicated server. Only a shared account on substandard > hardware, incapable of running PHP scripts. :( And several people have offered over the last few months, including myself. I am thinking about a dual xeon with 1 GB RAM that has a load average of 0.01... Nightly backups too, with disk a non-factor. I asked about bandwidth usage estimates but never got that far :-( This I simply do not understand. If there is a need for dedicated hardware, and it is being offered, then why is it not being accepted? Didn't CommandPrompt also offer dedicated hardware? Not wanting to start another heated debate, just wanting to understand what the deal is - as this topic keeps getting regurgitated without a conclusion. -- Mitch