Thread: Memory requirements for web-project

Memory requirements for web-project

From
Boris
Date:
Hello pgsql-general,

i need to calculate the memory requirement if i am using apache+pgsql.

Lets assume that i want 160.000 hits a day and pgsql takes 3 seconds
to work for each client, how much ram is required specially for pgsql?

I think I have to calculate the memory requiremend for a specific time
period.

160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.

lets round it up to 2 hits per secound. So I calculate now
2 hits per second * 3 seconds for a client
= 6 processes at the same time,

is that correct?

If so, how much is the memory requiremend? I have seen with "TOP" that
pgsql requires 5000kb for a request, but it is used per client (new processes for
each new client-request?)


--
Boris
www.x-itec.de



Re: Memory requirements for web-project

From
Lincoln Yeoh
Date:
At 05:10 PM 2/4/01 +0100, Boris wrote:
>160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
>Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.
>
>lets round it up to 2 hits per secound. So I calculate now
>2 hits per second * 3 seconds for a client
>= 6 processes at the same time,
>
>is that correct?

My guess is that your hits are likely to bunch up at certain times, rather
than be spread out evenly (90% of the people behave similarly 90% of the
time). So you might wish to multiply by 5 or 10 (or whatever you pluck from
the air ;) ) to cater for peak load.

Why are you saying it takes 3 seconds for a client? If it's because the
client is slow, you can do something about that, but if it's because your
query takes 3 seconds then maybe some people here could help with your query.

>If so, how much is the memory requiremend? I have seen with "TOP" that
>pgsql requires 5000kb for a request, but it is used per client (new
processes

The minimum size varies depending on what platform it's running on.

The max seems to vaguely depend on the size of the result set. But usually
most of the mem is reclaimed after the query is finished, at least on my
system.

Just do some tests with your bigger queries.

I'd say go for 512MB RAM.

Coz, usually you can't add RAM during peak load, and if you can, it usually
means you have hardware where 1GB or more is minimum anyway ;).

And you might be able to stick half on another server (web+app) if
desperate :).

Cheerio,
Link.


Re: Memory requirements for web-project

From
GH
Date:
On Sun, Feb 04, 2001 at 05:10:21PM +0100, some SMTP stream spewed forth:
> Hello pgsql-general,
>
> i need to calculate the memory requirement if i am using apache+pgsql.
>
> Lets assume that i want 160.000 hits a day and pgsql takes 3 seconds
> to work for each client, how much ram is required specially for pgsql?
>
> I think I have to calculate the memory requiremend for a specific time
> period.
>
> 160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
> Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.

Is it safe to preume that most of those 160,000 hits will come during
some relatively short peak period? You could end up with 5-10 hits per
second depending on the usage trends.

>
> lets round it up to 2 hits per secound. So I calculate now
> 2 hits per second * 3 seconds for a client
> = 6 processes at the same time,
>
> is that correct?

It seems correct to me.
>
> If so, how much is the memory requiremend? I have seen with "TOP" that
> pgsql requires 5000kb for a request, but it is used per client (new processes for
> each new client-request?)

Required memory would include Apache plus any modules (PHP, mod_perl,
cgis, etc.), postgres executible *and* whatever memory postgres requires for
processing the query (this may depend on what the user is doing...the
"type" of hit). Don't forget about caches and shared memory also.

gh

>
>
> --
> Boris
> www.x-itec.de
>
>

Re[2]: Memory requirements for web-project

From
Boris
Date:
Hello Lincoln,

Sunday, February 04, 2001, 5:51:37 PM, you wrote:

LY> At 05:10 PM 2/4/01 +0100, Boris wrote:
>>160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
>>Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.
>>
>>lets round it up to 2 hits per secound. So I calculate now
>>2 hits per second * 3 seconds for a client
>>= 6 processes at the same time,
>>
>>is that correct?

LY> My guess is that your hits are likely to bunch up at certain times, rather
LY> than be spread out evenly (90% of the people behave similarly 90% of the
LY> time). So you might wish to multiply by 5 or 10 (or whatever you pluck from
LY> the air ;) ) to cater for peak load.

LY> Why are you saying it takes 3 seconds for a client? If it's because the
LY> client is slow, you can do something about that, but if it's because your
LY> query takes 3 seconds then maybe some people here could help with your query.

>>If so, how much is the memory requiremend? I have seen with "TOP" that
>>pgsql requires 5000kb for a request, but it is used per client (new
LY> processes

LY> The minimum size varies depending on what platform it's running on.

LY> The max seems to vaguely depend on the size of the result set. But usually
LY> most of the mem is reclaimed after the query is finished, at least on my
LY> system.

LY> Just do some tests with your bigger queries.

LY> I'd say go for 512MB RAM.

LY> Coz, usually you can't add RAM during peak load, and if you can, it usually
LY> means you have hardware where 1GB or more is minimum anyway ;).

LY> And you might be able to stick half on another server (web+app) if
LY> desperate :).

LY> Cheerio,
LY> Link.

Hello Lincoln,

Sunday, February 04, 2001, 5:51:37 PM, you wrote:

LY> At 05:10 PM 2/4/01 +0100, Boris wrote:
>>160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
>>Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.
>>
>>lets round it up to 2 hits per secound. So I calculate now
>>2 hits per second * 3 seconds for a client
>>= 6 processes at the same time,
>>
>>is that correct?

LY> My guess is that your hits are likely to bunch up at certain times, rather
LY> than be spread out evenly (90% of the people behave similarly 90% of the
LY> time). So you might wish to multiply by 5 or 10 (or whatever you pluck from
LY> the air ;) ) to cater for peak load.

Ah, very interesting, yes - I forgot!

LY> Why are you saying it takes 3 seconds for a client? If it's because the

No its a server problem, has nothing to do with the database -)

LY> The minimum size varies depending on what platform it's running on.

Aha, interesting.

LY> The max seems to vaguely depend on the size of the result set. But usually
LY> most of the mem is reclaimed after the query is finished, at least on my
LY> system.

Interesting to know.

LY> Just do some tests with your bigger queries.

Ok, good idea.

LY> I'd say go for 512MB RAM.

That sounds good, the only question left is the memory requiremend of
apache per client, i do not completely understand the spawning things
with Apache. On high load there are alway minimum 10 processes left,
but where is the limit? Interesting thing.

LY> Coz, usually you can't add RAM during peak load, and if you can, it usually
LY> means you have hardware where 1GB or more is minimum anyway ;).

LY> And you might be able to stick half on another server (web+app) if
LY> desperate :).

ok.

LY> Cheerio,
LY> Link.

Thanks, that helped a lot!

--
Boris
www.x-itec.de



--
Boris [MCSE, CNA]
...................................................................
 X-ITEC : Consulting * Programming * Net-Security * Crypto-Research
........: [PRIVATE ADDRESS:]
        : Boris Köster eMail koester@x-itec.de http://www.x-itec.de
        : Grüne 33-57368 Lennestadt Germany Tel: +49 (0)2721 989400
        : 101  PERFECTION - SECURITY - STABILITY - FUNCTIONALITY
........:..........................................................

Everything I am writing is (c) by Boris Köster and may not be
rewritten or distributed in any way without my permission.



Re[2]: Memory requirements for web-project

From
Boris
Date:
Hello GH,

Sunday, February 04, 2001, 5:46:54 PM, you wrote:

G> On Sun, Feb 04, 2001 at 05:10:21PM +0100, some SMTP stream spewed forth:


G> Is it safe to preume that most of those 160,000 hits will come during
G> some relatively short peak period? You could end up with 5-10 hits per
G> second depending on the usage trends.

Yes I understand.

G> "type" of hit). Don't forget about caches and shared memory also.

What do you mean with the cache and the shared memory?


--
Boris



Re: Memory requirements for web-project

From
GH
Date:
On Sun, Feb 04, 2001 at 06:15:46PM +0100, some SMTP stream spewed forth:
> Hello GH,
>
> Sunday, February 04, 2001, 5:46:54 PM, you wrote:
>
> G> On Sun, Feb 04, 2001 at 05:10:21PM +0100, some SMTP stream spewed forth:
>
>
> G> Is it safe to preume that most of those 160,000 hits will come during
> G> some relatively short peak period? You could end up with 5-10 hits per
> G> second depending on the usage trends.
>
> Yes I understand.
>
> G> "type" of hit). Don't forget about caches and shared memory also.
>
> What do you mean with the cache and the shared memory?

PostgreSQL caches some of the data from the database. If you, for
example, wanted to have most of your database remain in memory, for speed,
then you would want to have more memory than the minimum needed to
handle the queries.

I meant that you would want to be sure that PostgreSQL is configured
with regard to the expected load and the number and mem requirements of
queries. You might see out of memory errors under load.


gh

>
>
> --
> Boris
>
>

Re: Memory requirements for web-project

From
"Mitch Vincent"
Date:
It depends on what you're doing per "hit"... I could allocate a hell of a
lot of memory in 3 seconds :-)

Buy twice as much RAM as you think you'll need, that way you're safe! I'd
get as much RAM as possible anyway, especially now that it's as cheap as it
is... 1 gig will only set you back several hundred dollars..

-Mitch

----- Original Message -----
From: "Boris" <koester@x-itec.de>
To: <pgsql-general@postgresql.org>
Sent: Sunday, February 04, 2001 11:10 AM
Subject: Memory requirements for web-project


> Hello pgsql-general,
>
> i need to calculate the memory requirement if i am using apache+pgsql.
>
> Lets assume that i want 160.000 hits a day and pgsql takes 3 seconds
> to work for each client, how much ram is required specially for pgsql?
>
> I think I have to calculate the memory requiremend for a specific time
> period.
>
> 160.000 hits are wanted, so I calculate 160.000 div 24 = approx 6666
> Hits a hour, div 60 = 111 Hits a minute, = 1,85 Hits per secound.
>
> lets round it up to 2 hits per secound. So I calculate now
> 2 hits per second * 3 seconds for a client
> = 6 processes at the same time,
>
> is that correct?
>
> If so, how much is the memory requiremend? I have seen with "TOP" that
> pgsql requires 5000kb for a request, but it is used per client (new
processes for
> each new client-request?)
>
>
> --
> Boris
> www.x-itec.de
>
>
>


Re: Re[2]: Memory requirements for web-project

From
Lincoln Yeoh
Date:
At 06:13 PM 04-02-2001 +0100, Boris wrote:
>LY> My guess is that your hits are likely to bunch up at certain times,
rather
>LY> than be spread out evenly (90% of the people behave similarly 90% of the
>LY> time). So you might wish to multiply by 5 or 10 (or whatever you pluck
from
>LY> the air ;) ) to cater for peak load.
>
>Ah, very interesting, yes - I forgot!

The peak vs the average ratio depends on the sort of application. It may
not vary that much for corporate data entry stuff.

>LY> Why are you saying it takes 3 seconds for a client? If it's because the
>
>No its a server problem, has nothing to do with the database -)

That's interesting. Any reason why 3 secs, which you can tell us?

>That sounds good, the only question left is the memory requiremend of
>apache per client, i do not completely understand the spawning things
>with Apache. On high load there are alway minimum 10 processes left,
>but where is the limit? Interesting thing.

Are you running apache and your web application on the same server as your
postgresql server? This would mean the total mem taken would probably be
apache+webapp+postgresql x max concurrent connections.

Apache 1.x starts up a number of processes to handle requests. Check out
what is said in httpd.conf about MinSpareServers and MaxSpareServers.

The max limit is controlled by MaxClients in httpd.conf

If it's a dedicated server for performance reasons you could have your
MaxSpareServers and StartServers reasonably high, that way Apache doesn't
have to keep killing and restarting new processes. Your MaxClients should
be kept to a value X where X concurrent connections won't cause your server
to swap.

e.g. example for peak if everything running on one server
Assuming 512MB RAM
apache=3MB with 2MB shared = 1MB per additional connection.
webapp=5MB with 2MB shared = 3MB per additional connection.
postgresql=6MB with 4MB shared (peak) = 2MB per additional connection.

Total = 6MB per additional connection.

So with MaxClients 60, you're limited to 60 http connections = 360MB,
leaving you 150MB to run various stuff, cope with strangeness, cache your
DB etc and let you sleep soundly at night.

512MB is ok for a DB only server doing your proposed load, but may be a bit
of a squeeze if you chuck your webserver and apps on it as well.

Cheerio,
Link.


Re: Re[2]: Memory requirements for web-project

From
Francisco Reyes
Date:
On Sun, 4 Feb 2001, Boris wrote:
> That sounds good, the only question left is the memory requiremend of
> apache per client, i do not completely understand the spawning things
> with Apache. On high load there are alway minimum 10 processes left,
> but where is the limit? Interesting thing.

I would like to recommend you use two machines. One for Apache/PHP and
another for the database. This would not only give you better performance,
but you would be able to scale where you need it without having to account
for two sides of the equation... in other words one of those two set of
operations (web vs DB) may be much higher of a bottleneck than the other.
Depending on your type of site and how heavy you use the DB you may even
find that the DB can catch up on a moderate setup while the apache/PHP
needs more hardware... or it could be totally the other way around.

It really has everything to do with your site and what it does.
How about telling you about the type of site and how will the DB be
involved in it.


Re: Memory requirements for web-project

From
Richard Huxton
Date:
Other people have said a lot on this, but IMHO missed some key points

Boris wrote:
>
> Hello pgsql-general,
>
> i need to calculate the memory requirement if i am using apache+pgsql.
>
> Lets assume that i want 160.000 hits a day and pgsql takes 3 seconds
> to work for each client, how much ram is required specially for pgsql?

As said in another case - depends upon the size of the query's result set.

> lets round it up to 2 hits per secound. So I calculate now
> 2 hits per second * 3 seconds for a client
> = 6 processes at the same time,
> is that correct?

No - as stated elsewhere, you need to allow for traffic peaks, so if you
have traffic logs examine them, otherwise pick your favourite number and
double it.
The other thing is that the time it takes to complete a query will
change the more clients you have running at the same time. Running six
simultaneous queries, you might get *one* result back in 3 secs, an
average of 6 secs and a maximum of 20 seconds for the slowest client.

Test the system with the "ab" apache benchmark tool (comes with apache)
or similar - this will give you a clearer idea of what too expect.

Then, of course, the result set will be stored in your perl/php/whatever
script before being sent to the client. You need to allow for that.

Also allow for the time to send the data to the client. Sending 100KB to
a client on a 28.8 modem can take a few seconds.

>
> If so, how much is the memory requiremend? I have seen with "TOP" that
> pgsql requires 5000kb for a request, but it is used per client (new processes for
> each new client-request?)

Don't forget to subtract the shared memory from each of these clients -
there will only be one copy of the code in memory for all the clients.

Bottom line - if money is tight, do as much testing as you can,
otherwise buy as much RAM as your system can take - too much tends not
to do a lot of damage.

- Richard Huxton