Thread: pgfoundry moved ...

pgfoundry moved ...

From
"Marc G. Fournier"
Date:
Should be a noticeable improvement ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
> Should be a noticeable improvement ...

Much better. A lot faster. *but*. The initial load time of each page is
still pretty bad. Meaning it takes a long time for it to start sending
the data back to the browser, but once it starts sending it's fast.

(Though initial time is also a *lot* faster than before, that's the part
that's still noticeable)

//Magnus

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
Ya, Joshua mentioned this ... I still have a bunch of work to do now that
the new server is in place, and I suspect the 'load times' you are seeing
now is database related, which is one aspect I'm going to be working on
over the next few days to improve ...

On Thu, 28 Apr 2005, Magnus Hagander wrote:

>> Should be a noticeable improvement ...
>
> Much better. A lot faster. *but*. The initial load time of each page is
> still pretty bad. Meaning it takes a long time for it to start sending
> the data back to the browser, but once it starts sending it's fast.
>
> (Though initial time is also a *lot* faster than before, that's the part
> that's still noticeable)
>
> //Magnus
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
Josh Berkus
Date:
Marc,

> Ya, Joshua mentioned this ... I still have a bunch of work to do now that
> the new server is in place, and I suspect the 'load times' you are seeing
> now is database related, which is one aspect I'm going to be working on
> over the next few days to improve ...

Hopefully we'll be moving pgFoundry off to the new server very soon.   So
don't devote a bunch of time towards it that could be used for fixing
performance on www.postgresql.org.

--
Josh Berkus
Aglio Database Solutions
San Francisco

Re: pgfoundry moved ...

From
"Gavin M. Roy"
Date:
If there are performance issues on www.postgresql.org it seems silly not
to use borg.  The machine is way under utilized and you have more than
enough donated bandwidth to play with there.

Gavin

Josh Berkus wrote:

>Marc,
>
>
>
>>Ya, Joshua mentioned this ... I still have a bunch of work to do now that
>>the new server is in place, and I suspect the 'load times' you are seeing
>>now is database related, which is one aspect I'm going to be working on
>>over the next few days to improve ...
>>
>>
>
>Hopefully we'll be moving pgFoundry off to the new server very soon.   So
>don't devote a bunch of time towards it that could be used for fixing
>performance on www.postgresql.org.
>
>
>


Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
> If there are performance issues on www.postgresql.org it
> seems silly not to use borg.  The machine is way under
> utilized and you have more than enough donated bandwidth to
> play with there.

The only performance issues related to the base www site right now I
think is the fact that I/O is horribly slow in the jail for svr2 (which
runs wwwmaster) and that one is already on borg IIRC. The mirror script
runs a *lot* faster on a heavily loaded linux server that I've tested it
on. Not sure if it's the I/O bw on borg that's over the edge, or if it's
the jails themselves causing it. I know Dave talked about comparing I/O
on the main host vs the jails, not sure if he ever got through to doing
it.


//Magnus

Re: pgfoundry moved ...

From
"Gavin M. Roy"
Date:
Hmm, that is interesting.  I have no experience in troubleshooting
jails, but that server was a pgsql db server (on Gentoo Linux) prior to
being borg, and has been very quick under heavy loads without any I/O
issues.

Gavin

Magnus Hagander wrote:

>>If there are performance issues on www.postgresql.org it
>>seems silly not to use borg.  The machine is way under
>>utilized and you have more than enough donated bandwidth to
>>play with there.
>>
>>
>
>The only performance issues related to the base www site right now I
>think is the fact that I/O is horribly slow in the jail for svr2 (which
>runs wwwmaster) and that one is already on borg IIRC. The mirror script
>runs a *lot* faster on a heavily loaded linux server that I've tested it
>on. Not sure if it's the I/O bw on borg that's over the edge, or if it's
>the jails themselves causing it. I know Dave talked about comparing I/O
>on the main host vs the jails, not sure if he ever got through to doing
>it.
>
>
>//Magnus
>
>---------------------------(end of broadcast)---------------------------
>TIP 9: the planner will ignore your desire to choose an index scan if your
>      joining column's datatypes do not match
>
>


Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Thu, 28 Apr 2005, Josh Berkus wrote:

> Marc,
>
>> Ya, Joshua mentioned this ... I still have a bunch of work to do now that
>> the new server is in place, and I suspect the 'load times' you are seeing
>> now is database related, which is one aspect I'm going to be working on
>> over the next few days to improve ...
>
> Hopefully we'll be moving pgFoundry off to the new server very soon.   So
> don't devote a bunch of time towards it that could be used for fixing
> performance on www.postgresql.org.

Talk to Magnus about that ... he's the one maintaining/running that server
...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
It is running on borg:

svr2# uname -a
FreeBSD svr2.postgresql.org 4.11-PRERELEASE FreeBSD 4.11-PRERELEASE #0: Thu Dec 16 10:52:33 PST 2004
root@borg:/usr/obj/usr/src/sys/GENERIC i386 


On Thu, 28 Apr 2005, Gavin M. Roy wrote:

> If there are performance issues on www.postgresql.org it seems silly not to
> use borg.  The machine is way under utilized and you have more than enough
> donated bandwidth to play with there.
>
> Gavin
>
> Josh Berkus wrote:
>
>> Marc,
>>
>>
>>> Ya, Joshua mentioned this ... I still have a bunch of work to do now that
>>> the new server is in place, and I suspect the 'load times' you are seeing
>>> now is database related, which is one aspect I'm going to be working on
>>> over the next few days to improve ...
>>>
>>
>> Hopefully we'll be moving pgFoundry off to the new server very soon.   So
>> don't devote a bunch of time towards it that could be used for fixing
>> performance on www.postgresql.org.
>>
>>
>
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Thu, 28 Apr 2005, Gavin M. Roy wrote:

> Hmm, that is interesting.  I have no experience in troubleshooting jails, but
> that server was a pgsql db server (on Gentoo Linux) prior to being borg, and
> has been very quick under heavy loads without any I/O issues.

I just removed svr2 as being a relay point for outgoing list emails ...
looking at iostat 5 on that machine, the CPU was wrong close to 0 idle,
whereas now seems to be closer to 100 ... might have been throwing too
much into the works ;(

Does that make a difference for anyone?


  > > Gavin
>
> Magnus Hagander wrote:
>
>>> If there are performance issues on www.postgresql.org it seems silly not
>>> to use borg.  The machine is way under utilized and you have more than
>>> enough donated bandwidth to play with there.
>>>
>>
>> The only performance issues related to the base www site right now I
>> think is the fact that I/O is horribly slow in the jail for svr2 (which
>> runs wwwmaster) and that one is already on borg IIRC. The mirror script
>> runs a *lot* faster on a heavily loaded linux server that I've tested it
>> on. Not sure if it's the I/O bw on borg that's over the edge, or if it's
>> the jails themselves causing it. I know Dave talked about comparing I/O
>> on the main host vs the jails, not sure if he ever got through to doing
>> it.
>>
>>
>> //Magnus
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 9: the planner will ignore your desire to choose an index scan if your
>>      joining column's datatypes do not match
>>
>
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
>> Hmm, that is interesting.  I have no experience in
>troubleshooting jails, but
>> that server was a pgsql db server (on Gentoo Linux) prior to
>being borg, and
>> has been very quick under heavy loads without any I/O issues.
>
>I just removed svr2 as being a relay point for outgoing list
>emails ...
>looking at iostat 5 on that machine, the CPU was wrong close
>to 0 idle,
>whereas now seems to be closer to 100 ... might have been throwing too
>much into the works ;(
>
>Does that make a difference for anyone?

It *feels* a bit faster, but I don't have the time to run an actual
mirror test script to check right now (will try to do that later). And
mail relaying in general tends to load up the I/O quite a bit (lots of
small syncs in different areas of the disk)

I'm not sure what kind of hardware it's on. I did the following totally
unscientific benchmark to get a quick figure. The comparison box is a
three years old HP Proliant DL360 with 512Mb RAM and a RAID-1 disk set,
that's loaded with a whole bunch of I/O intensive stuff (like mail
relaying for about 20,000 users, several web hosts, some db etc). it
runs linux 2.4.30 with an ext3 filesystem (totally untuned).


Reference box:
real    0m21.318s
user    0m6.160s
sys     0m17.270s

borg:
real    0m25.795s
user    0m2.318s
sys     0m20.453s

svr2:
real    0m33.068s
user    0m2.226s
sys     0m21.130s


From this test it seems the jail has some overhead, around 20-25%. Not
sure how to read the difference witht he reference box considering I
don't know what hw is in Borg :) But as I said, it does *seem* a lot
snappier now.

//Magnus

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Thu, 28 Apr 2005, Magnus Hagander wrote:

>>> Hmm, that is interesting.  I have no experience in
>> troubleshooting jails, but
>>> that server was a pgsql db server (on Gentoo Linux) prior to
>> being borg, and
>>> has been very quick under heavy loads without any I/O issues.
>>
>> I just removed svr2 as being a relay point for outgoing list
>> emails ...
>> looking at iostat 5 on that machine, the CPU was wrong close
>> to 0 idle,
>> whereas now seems to be closer to 100 ... might have been throwing too
>> much into the works ;(
>>
>> Does that make a difference for anyone?
>
> It *feels* a bit faster, but I don't have the time to run an actual
> mirror test script to check right now (will try to do that later). And
> mail relaying in general tends to load up the I/O quite a bit (lots of
> small syncs in different areas of the disk)
>
> I'm not sure what kind of hardware it's on. I did the following totally
> unscientific benchmark to get a quick figure. The comparison box is a
> three years old HP Proliant DL360 with 512Mb RAM and a RAID-1 disk set,
> that's loaded with a whole bunch of I/O intensive stuff (like mail
> relaying for about 20,000 users, several web hosts, some db etc). it
> runs linux 2.4.30 with an ext3 filesystem (totally untuned).
>
>
> Reference box:
> real    0m21.318s
> user    0m6.160s
> sys     0m17.270s
>
> borg:
> real    0m25.795s
> user    0m2.318s
> sys     0m20.453s
>
> svr2:
> real    0m33.068s
> user    0m2.226s
> sys     0m21.130s
>
>
> From this test it seems the jail has some overhead, around 20-25%. Not
> sure how to read the difference witht he reference box considering I
> don't know what hw is in Borg :) But as I said, it does *seem* a lot
> snappier now.

The overhead that is being experienced is most likely the unionfs itself,
and not the jail ... its one of the reasons I'm watching the VFS rewrite
work being done by DragonFlyBSD closely, since they plan on fixing the
bugs in unionfs, including improving its overall performance ... :(

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Dave Page"
Date:

> -----Original Message-----
> From: pgsql-www-owner@postgresql.org
> [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Marc G. Fournier
> Sent: 28 April 2005 19:43
> To: Gavin M. Roy
> Cc: Magnus Hagander; Josh Berkus; Marc G. Fournier;
> pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] pgfoundry moved ...
>
> I just removed svr2 as being a relay point for outgoing list
> emails ...
> looking at iostat 5 on that machine, the CPU was wrong close
> to 0 idle,
> whereas now seems to be closer to 100 ... might have been
> throwing too
> much into the works ;(
>
> Does that make a difference for anyone?

Haven't checked that yet, but if you need a replacment relay, use
ferengi - theres a dual 3GHz Xeon box doing nothing more than serving
1/5th of the static content for www.postgresql.org.

Borg is fairly loaded in comparison. It's a dual P3 running one of the 5
static servers, as well as the master dynamic server and it's PostgreSQL
backend.

Regards, Dave.

Re: pgfoundry moved ...

From
"Dave Page"
Date:

> -----Original Message-----
> From: pgsql-www-owner@postgresql.org
> [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Magnus Hagander
> Sent: 28 April 2005 18:13
> To: Gavin M. Roy; Josh Berkus
> Cc: Marc G. Fournier; pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] pgfoundry moved ...
>
> I know Dave talked about
> comparing I/O
> on the main host vs the jails, not sure if he ever got
> through to doing
> it.

I ran Bonnie++ a few times and saw very variable performance. Both the
host and svr2 peaked about the same as I recall though.

As soon as we have another front end webserver, I do want to remove both
svr2 and borg from the rotation. Both are very overworked compared to
the other boxes.

Also, Magnus was say that the bt processes on svr4 are very sluggish.
Perhaps we should move them to ferengi as well.

Regards, Dave.

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Thu, 28 Apr 2005, Dave Page wrote:

> Haven't checked that yet, but if you need a replacment relay, use
> ferengi - theres a dual 3GHz Xeon box doing nothing more than serving
> 1/5th of the static content for www.postgresql.org.

That would be cool ... we've got two relays right now, but spreading the
load does help by keeping the queues shortened :)

Can you configure it to accept connections from svr1.postgresql.org
(200.46.204.71)?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Dave Page"
Date:

> -----Original Message-----
> From: Marc G. Fournier [mailto:scrappy@postgresql.org]
> Sent: 28 April 2005 20:46
> To: Dave Page
> Cc: Marc G. Fournier; Gavin M. Roy; Magnus Hagander; Josh
> Berkus; pgsql-www@postgresql.org
> Subject: RE: [pgsql-www] pgfoundry moved ...
>
> On Thu, 28 Apr 2005, Dave Page wrote:
>
> > Haven't checked that yet, but if you need a replacment relay, use
> > ferengi - theres a dual 3GHz Xeon box doing nothing more
> than serving
> > 1/5th of the static content for www.postgresql.org.
>
> That would be cool ... we've got two relays right now, but
> spreading the
> load does help by keeping the queues shortened :)
>
> Can you configure it to accept connections from svr1.postgresql.org
> (200.46.204.71)?

OK, well having beaten postfix and suse firewall into shape that's ready
to go now.

Regards, Dave.

Re: pgfoundry moved ...

From
Oleg Bartunov
Date:
Is't possible to understand what's an actual problem, database or
web part ? Is't possible to see timings for typical longest queries ?
Probably there is some profiling support which show timings for
each component used. If gbord would be Mason based applications it could be
done very easy.

Oleg
On Thu, 28 Apr 2005, Dave Page wrote:

>
>
>> -----Original Message-----
>> From: pgsql-www-owner@postgresql.org
>> [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Marc G. Fournier
>> Sent: 28 April 2005 19:43
>> To: Gavin M. Roy
>> Cc: Magnus Hagander; Josh Berkus; Marc G. Fournier;
>> pgsql-www@postgresql.org
>> Subject: Re: [pgsql-www] pgfoundry moved ...
>>
>> I just removed svr2 as being a relay point for outgoing list
>> emails ...
>> looking at iostat 5 on that machine, the CPU was wrong close
>> to 0 idle,
>> whereas now seems to be closer to 100 ... might have been
>> throwing too
>> much into the works ;(
>>
>> Does that make a difference for anyone?
>
> Haven't checked that yet, but if you need a replacment relay, use
> ferengi - theres a dual 3GHz Xeon box doing nothing more than serving
> 1/5th of the static content for www.postgresql.org.
>
> Borg is fairly loaded in comparison. It's a dual P3 running one of the 5
> static servers, as well as the master dynamic server and it's PostgreSQL
> backend.
>
> Regards, Dave.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>               http://www.postgresql.org/docs/faq
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: pgfoundry moved ...

From
"Dave Page"
Date:

> -----Original Message-----
> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
> Sent: 29 April 2005 05:42
> To: Dave Page
> Cc: Marc G. Fournier; Gavin M. Roy; Magnus Hagander; Josh
> Berkus; pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] pgfoundry moved ...
>
> Is't possible to understand what's an actual problem, database or
> web part ? Is't possible to see timings for typical longest queries ?
> Probably there is some profiling support which show timings for
> each component used. If gbord would be Mason based
> applications it could be
> done very easy.

We've spent time on that in the past, and nothing obvious is apparent,
other than disk IO being slow in general. The same problem was seen when
svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
issue.

Regards, Dave.

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Fri, 29 Apr 2005, Dave Page wrote:

>
>
>> -----Original Message-----
>> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
>> Sent: 29 April 2005 05:42
>> To: Dave Page
>> Cc: Marc G. Fournier; Gavin M. Roy; Magnus Hagander; Josh
>> Berkus; pgsql-www@postgresql.org
>> Subject: Re: [pgsql-www] pgfoundry moved ...
>>
>> Is't possible to understand what's an actual problem, database or
>> web part ? Is't possible to see timings for typical longest queries ?
>> Probably there is some profiling support which show timings for
>> each component used. If gbord would be Mason based
>> applications it could be
>> done very easy.
>
> We've spent time on that in the past, and nothing obvious is apparent,
> other than disk IO being slow in general. The same problem was seen when
> svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
> issue.

The only thing that bothers me is that it shouldn't be as apparent on borg
as it was on ours, since there is only *one* VM running there, vs >30 that
were running on our servers ... so unionfs shouldn't be hampering
performance nearly as much ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
>> Is't possible to understand what's an actual problem, database or
>> web part ? Is't possible to see timings for typical longest queries ?
>> Probably there is some profiling support which show timings for
>> each component used. If gbord would be Mason based
>> applications it could be
>> done very easy.
>
>We've spent time on that in the past, and nothing obvious is apparent,
>other than disk IO being slow in general. The same problem was
>seen when
>svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
>issue.

In my experience, it's at least definitly not the db. Pages that have
nothing to do with the db has been equally slow. So I'm willing to buy
in with Daves guess.

//Magnus

Re: pgfoundry moved ...

From
"Dave Page"
Date:

> -----Original Message-----
> From: Marc G. Fournier [mailto:scrappy@postgresql.org]
> Sent: 29 April 2005 15:35
> To: Dave Page
> Cc: Oleg Bartunov; Marc G. Fournier; Gavin M. Roy; Magnus
> Hagander; Josh Berkus; pgsql-www@postgresql.org
> Subject: RE: [pgsql-www] pgfoundry moved ...
>
> The only thing that bothers me is that it shouldn't be as
> apparent on borg
> as it was on ours, since there is only *one* VM running
> there, vs >30 that
> were running on our servers ... so unionfs shouldn't be hampering
> performance nearly as much ...

Agreed, however, that's a dual PIII 1GHz box with multi-gigabytes of RAM
and RAID. I can say with confidence that I've seen better responsiveness
and seemingly higher performance on Linux boxes running at half that
speed with a quarter of the memory. The main site build takes ages on
there, much as it did on svr2 - running it here on what is effectively a
P4 desktop takes about 20 minutes on a bad day.

Regards, Dave

Re: pgfoundry moved ...

From
Oleg Bartunov
Date:
On Fri, 29 Apr 2005, Magnus Hagander wrote:

>>> Is't possible to understand what's an actual problem, database or
>>> web part ? Is't possible to see timings for typical longest queries ?
>>> Probably there is some profiling support which show timings for
>>> each component used. If gbord would be Mason based
>>> applications it could be
>>> done very easy.
>>
>> We've spent time on that in the past, and nothing obvious is apparent,
>> other than disk IO being slow in general. The same problem was
>> seen when
>> svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
>> issue.
>
> In my experience, it's at least definitly not the db. Pages that have
> nothing to do with the db has been equally slow. So I'm willing to buy
> in with Daves guess.

Ok, I think it's bad  architecture. I already told Marc about that.
It's very easy to separate processing binary static files like images from
dynamic content. Just setup thttpd. Next step to setup frontend web server
which should be very light with cacheing capability - we use mod_accel module
for apache. It's frontentd which communicate with browser (probably slow link).
Backend, with PHP, mod_auth_pgsql should be use for page generation, it
should communicate only with frontend. The main idea is that you need much
less backends (plus postgres connection for each backend), so much lesser
resources will be used.  What's the problem to setup this ?

>
> //Magnus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>      subscribe-nomail command to majordomo@postgresql.org so that your
>      message can get through to the mailing list cleanly
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
>>>> Is't possible to understand what's an actual problem, database or
>>>> web part ? Is't possible to see timings for typical
>longest queries ?
>>>> Probably there is some profiling support which show timings for
>>>> each component used. If gbord would be Mason based
>>>> applications it could be
>>>> done very easy.
>>>
>>> We've spent time on that in the past, and nothing obvious
>is apparent,
>>> other than disk IO being slow in general. The same problem was
>>> seen when
>>> svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
>>> issue.
>>
>> In my experience, it's at least definitly not the db. Pages that have
>> nothing to do with the db has been equally slow. So I'm
>willing to buy
>> in with Daves guess.
>
>Ok, I think it's bad  architecture. I already told Marc about that.
>It's very easy to separate processing binary static files like
>images from
>dynamic content. Just setup thttpd. Next step to setup
>frontend web server
>which should be very light with cacheing capability - we use
>mod_accel module
>for apache. It's frontentd which communicate with browser
>(probably slow link).

Um. You really aren't up to speed on how things are, are you? ;-) We
*do* use static frontends. Five of them actually, globally distributed.
This is not where the performance problem is.

>Backend, with PHP, mod_auth_pgsql should be use for page
>generation, it
>should communicate only with frontend. The main idea is that
>you need much
>less backends (plus postgres connection for each backend), so
>much lesser
>resources will be used.  What's the problem to setup this ?

Nothing - though a slightly different method is used where the frontends
get their static content with rsync. But the main idea is the same.

Except the backend is slow *anyway* - even with just the
regen-for-frontend stuff loading it down. It doesn't show up for the end
user, but it does make the site rebuild slower, whjich means it has to
run less frequently (once/hour for the most often updated stuff, less
often for docs and ftp tree).

//Magnus

Re: pgfoundry moved ...

From
Oleg Bartunov
Date:
On Fri, 29 Apr 2005, Magnus Hagander wrote:

>>>>> Is't possible to understand what's an actual problem, database or
>>>>> web part ? Is't possible to see timings for typical
>> longest queries ?
>>>>> Probably there is some profiling support which show timings for
>>>>> each component used. If gbord would be Mason based
>>>>> applications it could be
>>>>> done very easy.
>>>>
>>>> We've spent time on that in the past, and nothing obvious
>> is apparent,
>>>> other than disk IO being slow in general. The same problem was
>>>> seen when
>>>> svr2 was on one of Marc's boxes. I'm fairly convinced it's a unionfs
>>>> issue.
>>>
>>> In my experience, it's at least definitly not the db. Pages that have
>>> nothing to do with the db has been equally slow. So I'm
>> willing to buy
>>> in with Daves guess.
>>
>> Ok, I think it's bad  architecture. I already told Marc about that.
>> It's very easy to separate processing binary static files like
>> images from
>> dynamic content. Just setup thttpd. Next step to setup
>> frontend web server
>> which should be very light with cacheing capability - we use
>> mod_accel module
>> for apache. It's frontentd which communicate with browser
>> (probably slow link).
>
> Um. You really aren't up to speed on how things are, are you? ;-) We
> *do* use static frontends. Five of them actually, globally distributed.
> This is not where the performance problem is.

I see that your static frontend has PHP compiled and other stuff.

curl -I http://pgfoundry.org/
HTTP/1.1 200 OK
Date: Fri, 29 Apr 2005 19:04:14 GMT
Server: Apache/1.3.33 (Unix) mod_auth_pgsql/0.9.12 PHP/4.3.11
X-Powered-By: PHP/4.3.11
Content-Type: text/html; charset=UTF-8

it's silly. All that stuff eat resources. Even old pentium would
serve a lot of *static* pages without problem, you don't need "globally distributed"
frontends.

>
>> Backend, with PHP, mod_auth_pgsql should be use for page
>> generation, it
>> should communicate only with frontend. The main idea is that
>> you need much
>> less backends (plus postgres connection for each backend), so
>> much lesser
>> resources will be used.  What's the problem to setup this ?
>
> Nothing - though a slightly different method is used where the frontends
> get their static content with rsync. But the main idea is the same.
>
> Except the backend is slow *anyway* - even with just the
> regen-for-frontend stuff loading it down. It doesn't show up for the end
> user, but it does make the site rebuild slower, whjich means it has to
> run less frequently (once/hour for the most often updated stuff, less
> often for docs and ftp tree).

Hmm, not very nice. I don't think pgfoundry is very busy site.
Is there some web stat available ? How many pages generated dynamically ?
Could you run simple ab benchmark ? Technology I described is simple, commonly
adopted and proven in rather busy sites (millions visitors per day).
Anyway, I just tried to help. Probably, you know more information why
services are so slow and has clever idea how to improve situation.


>
> //Magnus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>               http://www.postgresql.org/docs/faq
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Sat, 30 Apr 2005, Oleg Bartunov wrote:

> Hmm, not very nice. I don't think pgfoundry is very busy site.

I thought we were talking about the main web server?  How did pgfoundry
come into this?

> Is there some web stat available ? How many pages generated dynamically ?

all of pgfoundry is generated dynamically ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Magnus Hagander"
Date:
>> Um. You really aren't up to speed on how things are, are you? ;-) We
>> *do* use static frontends. Five of them actually, globally
>distributed.
>> This is not where the performance problem is.
>
>I see that your static frontend has PHP compiled and other stuff.
>
>curl -I http://pgfoundry.org/
>HTTP/1.1 200 OK
>Date: Fri, 29 Apr 2005 19:04:14 GMT
>Server: Apache/1.3.33 (Unix) mod_auth_pgsql/0.9.12 PHP/4.3.11
>X-Powered-By: PHP/4.3.11
>Content-Type: text/html; charset=UTF-8

I was talking about www.postgresql.org, not pgfoundry.org. I realise the
subject still says pgfoundry, but the discussion was more along the
performance of www.postgresql.org at this stage..


>it's silly. All that stuff eat resources. Even old pentium would
>serve a lot of *static* pages without problem, you don't need
>"globally distributed"
>frontends.

As long as it's not used, it hardly makes a difference. Anyway, we need
the distribution mainly for redundancy, not performance.


>> Except the backend is slow *anyway* - even with just the
>> regen-for-frontend stuff loading it down. It doesn't show up
>for the end
>> user, but it does make the site rebuild slower, whjich means
>it has to
>> run less frequently (once/hour for the most often updated stuff, less
>> often for docs and ftp tree).
>
>Hmm, not very nice. I don't think pgfoundry is very busy site.
>Is there some web stat available ? How many pages generated
>dynamically ?
>Could you run simple ab benchmark ? Technology I described is
>simple, commonly
>adopted and proven in rather busy sites (millions visitors per day).
>Anyway, I just tried to help. Probably, you know more information why
>services are so slow and has clever idea how to improve situation.

Again, we're discussing differen things.
As for pgfoundry, yes, it seems to put a very high load on the systems,
but I can't talk abuot any details there, since I don't know them. I'm
sure there are many different ways to solve those issues.

//Magnus

Re: pgfoundry moved ...

From
Oleg Bartunov
Date:
On Fri, 29 Apr 2005, Marc G. Fournier wrote:

> On Sat, 30 Apr 2005, Oleg Bartunov wrote:
>
>> Hmm, not very nice. I don't think pgfoundry is very busy site.
>
> I thought we were talking about the main web server?  How did pgfoundry come
> into this?

see subject :)

>
>> Is there some web stat available ? How many pages generated dynamically ?
>
> all of pgfoundry is generated dynamically ...
>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: pgfoundry moved ...

From
Oleg Bartunov
Date:
ok, guys. we really talked about different things :)
The main web site is indeed fully static, so such slowdown usually
inidcates limiting resources (sockets, open files, ...).

Oleg

On Fri, 29 Apr 2005, Magnus Hagander wrote:

>>> Um. You really aren't up to speed on how things are, are you? ;-) We
>>> *do* use static frontends. Five of them actually, globally
>> distributed.
>>> This is not where the performance problem is.
>>
>> I see that your static frontend has PHP compiled and other stuff.
>>
>> curl -I http://pgfoundry.org/
>> HTTP/1.1 200 OK
>> Date: Fri, 29 Apr 2005 19:04:14 GMT
>> Server: Apache/1.3.33 (Unix) mod_auth_pgsql/0.9.12 PHP/4.3.11
>> X-Powered-By: PHP/4.3.11
>> Content-Type: text/html; charset=UTF-8
>
> I was talking about www.postgresql.org, not pgfoundry.org. I realise the
> subject still says pgfoundry, but the discussion was more along the
> performance of www.postgresql.org at this stage..
>
>
>> it's silly. All that stuff eat resources. Even old pentium would
>> serve a lot of *static* pages without problem, you don't need
>> "globally distributed"
>> frontends.
>
> As long as it's not used, it hardly makes a difference. Anyway, we need
> the distribution mainly for redundancy, not performance.
>
>
>>> Except the backend is slow *anyway* - even with just the
>>> regen-for-frontend stuff loading it down. It doesn't show up
>> for the end
>>> user, but it does make the site rebuild slower, whjich means
>> it has to
>>> run less frequently (once/hour for the most often updated stuff, less
>>> often for docs and ftp tree).
>>
>> Hmm, not very nice. I don't think pgfoundry is very busy site.
>> Is there some web stat available ? How many pages generated
>> dynamically ?
>> Could you run simple ab benchmark ? Technology I described is
>> simple, commonly
>> adopted and proven in rather busy sites (millions visitors per day).
>> Anyway, I just tried to help. Probably, you know more information why
>> services are so slow and has clever idea how to improve situation.
>
> Again, we're discussing differen things.
> As for pgfoundry, yes, it seems to put a very high load on the systems,
> but I can't talk abuot any details there, since I don't know them. I'm
> sure there are many different ways to solve those issues.
>
> //Magnus
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: pgfoundry moved ...

From
"Marc G. Fournier"
Date:
On Sat, 30 Apr 2005, Oleg Bartunov wrote:

> see subject :)

And it was meant to have been a simple "things should be better" note, and
it tangented off to the main web server :)

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: pgfoundry moved ...

From
"Dave Page"
Date:


-----Original Message-----
From: Oleg Bartunov [mailto:oleg@sai.msu.su]
Sent: Fri 4/29/2005 8:11 PM
To: Magnus Hagander
Cc: Dave Page; Marc G. Fournier; Gavin M. Roy; Josh Berkus; pgsql-www@postgresql.org
Subject: Re: [pgsql-www] pgfoundry moved ...

> Ok, I think it's bad  architecture. I already told Marc about that.
> It's very easy to separate processing binary static files like images from
> dynamic content. Just setup thttpd. Next step to setup frontend web server
> which should be very light with cacheing capability - we use mod_accel module
> for apache. It's frontentd which communicate with browser (probably slow link).
> Backend, with PHP, mod_auth_pgsql should be use for page generation, it
> should communicate only with frontend. The main idea is that you need much
> less backends (plus postgres connection for each backend), so much lesser
> resources will be used.  What's the problem to setup this ?

This is irrelevant to svr2.

Regards, Dave