Thread: pgfoundry is very slow !
I notice tracking why our crawler is slow, that pgfoundry generates pages for awfully long time ! here is excerption from crawler's log: 14658 17.04 23:47:40 IC: http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: no cached document, usingversion from server 14658 17.04 23:47:40 URL http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: 46.023 sec 14658 17.04 23:49:00 IC: http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: no cached document, usingversion from server 14658 17.04 23:49:00 URL http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: 78.923 sec Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
Oleg, > I notice tracking why our crawler is slow, that > pgfoundry generates pages for awfully long time ! > here is excerption from crawler's log: Within 2 weeks, we'll be moving pgFoundry to a new server where it will run behind Squid, which should solve this problem. However, this does explain why the googlebot is slamming us. -- --Josh Josh Berkus Aglio Database Solutions San Francisco
I don't know what it is about pgfoundry ... the server loadavg is load, and looking at vmstat 5 for CPU usage, idle time is high ... and, I have eacellerator setup with a 8 Meg of shared memory cache, and no disk caching happening, and its only using half of that ... if someone can suggest something else I can look at, please let me know and I'll happily look at it ... Oleg, GForge rip'd out the ability to easily set a seperate 'images' server from the code, so I've *tried* to use Apache's Rewrite rules to spread it over two seperate physical machines, but from what I can tell, even with the Rewrite, it still hits the first server before reading from the second ... if there is a better way I can do this, please feel free to point it out to me and I can try that also ... On Sun, 17 Apr 2005, Oleg Bartunov wrote: > I notice tracking why our crawler is slow, that > pgfoundry generates pages for awfully long time ! > here is excerption from crawler's log: > > 14658 17.04 23:47:40 IC: > http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: no > cached document, using version from server > 14658 17.04 23:47:40 URL > http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: > 46.023 sec > > 14658 17.04 23:49:00 IC: > http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: no > cached document, using version from server > 14658 17.04 23:49:00 URL > http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: > 78.923 sec > > Regards, > Oleg > _____________________________________________________________ > Oleg Bartunov, sci.researcher, hostmaster of AstroNet, > Sternberg Astronomical Institute, Moscow University (Russia) > Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ > phone: +007(095)939-16-83, +007(095)939-23-83 > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
On Sun, 17 Apr 2005, Marc G. Fournier wrote: > > > I don't know what it is about pgfoundry ... the server loadavg is load, and > looking at vmstat 5 for CPU usage, idle time is high ... and, I have > eacellerator setup with a 8 Meg of shared memory cache, and no disk caching > happening, and its only using half of that ... > > if someone can suggest something else I can look at, please let me know and > I'll happily look at it ... What's about database server ? Do you monitor long sql query ? > > Oleg, GForge rip'd out the ability to easily set a seperate 'images' server > from the code, so I've *tried* to use Apache's Rewrite rules to spread it > over two seperate physical machines, but from what I can tell, even with the > Rewrite, it still hits the first server before reading from the second ... if > there is a better way I can do this, please feel free to point it out to me > and I can try that also ... > Are you sure images are bottleneck ? In any case, i'd recommend frontend-backend architecture to let different servers do different jobs. frontend - simple apache with mod_accel (mod_proxy) support, it's light and fast and does only 2 things - serves images adn proxy dynamic requests to fat backend server which generates pages. With proper setup of backend (valid modification date and content-length returned) frontend even could cache pages. we use mod_accel (http://www.sysoev.ru/en/) mod_proxy setup described in apache documentation and you don't need touch gforge code. > > > > On Sun, 17 Apr 2005, Oleg Bartunov wrote: > >> I notice tracking why our crawler is slow, that >> pgfoundry generates pages for awfully long time ! >> here is excerption from crawler's log: >> >> 14658 17.04 23:47:40 IC: >> http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: no >> cached document, using version from server >> 14658 17.04 23:47:40 URL >> http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=266&add_cat=1: >> 46.023 sec >> >> 14658 17.04 23:49:00 IC: >> http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: no >> cached document, using version from server >> 14658 17.04 23:49:00 URL >> http://pgfoundry.org/tracker/admin/?group_id=1000048&atid=267&add_cat=1: >> 78.923 sec >> >> Regards, >> Oleg >> _____________________________________________________________ >> Oleg Bartunov, sci.researcher, hostmaster of AstroNet, >> Sternberg Astronomical Institute, Moscow University (Russia) >> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ >> phone: +007(095)939-16-83, +007(095)939-23-83 >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 3: if posting/reading through Usenet, please send an appropriate >> subscribe-nomail command to majordomo@postgresql.org so that your >> message can get through to the mailing list cleanly >> > > ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664 > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
On Sun, Apr 17, 2005 at 11:14:05PM -0300, Marc G. Fournier wrote: > > if someone can suggest something else I can look at, please let me know > and I'll happily look at it ... Have you done any process traces (strace, ktrace, truss, etc.) on the web server to see what it does when handling a request? -- Michael Fuhr http://www.fuhr.org/~mfuhr/
On Mon, 18 Apr 2005, Oleg Bartunov wrote: > What's about database server ? Do you monitor long sql query ? Hate to admit, but never thought about it ... trying that route now, but so far, I'm down to a log_min_duration_statement of 100, and nothing going to the logs ... down to 50, I'm starting to get a bit, but nothing right off that looks like pgfoudrny stuff, but will let the logs build up and scan through it in the morning ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664