Thread: Virtualization vs. sharing a server

1. Is there a configuration option we need to consider to share this server?
2. What is the recomended hardware for a PostgreSQL server that has to support between 500 and 1000 simultaneous user connections? Is there a whitepaper about this?

Ramiro,

I don't know why virtualization is considered a no-no.  We use VMware ESX.  On some smaller applications we run both the application and database on the virutal machine.  We've not had a issue with this combination in 5+ years.  We also have 6 images that run on 2 machines just for the appliations.  Each of these 5 machines is running Windows 2003 Server with Citrix XenAPP.  We do have 2 database servers that are running 2 Quad Cores Intel processors, one with 32 gig and one with 64 gig.  We also have a couple of old PIII 1 Gig machines running as our AD servers.

Each of the database servers are currently running SQL Anywhere 9 and MSSQL 2005.  We've never had a issue.  We keep one server (64 Gig one) as our primary server and the other as a hot backup with auto failover.

If it were not for the virtualized application machines we would have had to purchase quite a bit more hardware.  Since these are all quad core with 32 gig running Windows 2003 64 bit, we can run about 100 users concurrently on each application server before we start to see a strain.

In the past 5 years we've had one time when (when we only 2 physical servers) and we lost one and the other was able to handle all of the application needs although much more slowly.

I think that the success of virutalization is like anything else, having the right people to set it up.  We have no one in our compay to do this kind of thing so we contract it out to others with platinum certifications from the various vendors (Citrix/ Windows Servers and VMWare). 

When we finish our commerical application to production we will probably contract with EnterpriseDB to help us with tuning Postgres and the servers they run on.

Best Regards

Michael Gould wrote:
>
> I don't know why virtualization is considered a no-no...Since these
> are all quad core with 32 gig running Windows 2003 64 bit, we can run
> about 100 users concurrently on each application server before we
> start to see a strain.
>

You answered your own question here.  Ramiro is looking for suggestions
for how to scale up to >500 connections at once, and it's not that
likely virtualization can fill any useful role in that context.  If
you're happy with 100, sure you can deploy on VMware ESX and have that
work.  There are performance vs. manageability tradeoffs when deciding
if virtualized deployment makes sense, and for smaller workloads it's
easy to dismiss the performance side of things as not a limiting factor
and therefore favor VMs.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us

Ramiro Barreca wrote:
>
>    1. Is there a configuration option we need to consider to share
>       this server?
>

The two main configuration options that impact how much RAM PostgreSQL
uses are shared_buffers and work_mem.  If the server is shared, you just
need to avoid tuning those upwards as far as you would on a dedicated
system.  The defaults for both are tiny, so unless you've already pushed
these upwards a lot you're unlikely to have any issues in an initial
deployment.


>    1. What is the recomended hardware for a PostgreSQL server that has
>       to support between 500 and 1000 simultaneous user connections?
>       Is there a whitepaper about this?
>

The recommendation is "try not to do that".  PostgreSQL connections are
fairly expensive, and scaling into hundreds of connections on a single
system tends to break down due to connection creation/teardown
overhead.  You should be investigating a connection pooler to put in
between the application and the database, limiting the number of
connections to something much smaller.  PgBouncer is most people's first
choice for this scale of database:  http://pgfoundry.org/projects/pgbouncer

Once you've gotten past that issue, recommended hardware varies greatly
depending on workload.  Read vs. write balance?  Size of the critical
working set in RAM?  Size of the database?  Peak commit rate?  Types of
queries executed by the application?  It all factors into the decision.

Add in business requirements for reliability and the inevitable vendor
preference, you're likely to end up with a stack of possible
configurations you could consider to sort through.  There is no generic
recommendation for larger systems.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us

On Tue, March 30, 2010 06:09, Greg Smith wrote:
> Michael Gould wrote:
>>
>> I don't know why virtualization is considered a no-no...Since these
>> are all quad core with 32 gig running Windows 2003 64 bit, we can run
>> about 100 users concurrently on each application server before we
>> start to see a strain.
>>
>
> You answered your own question here.  Ramiro is looking for suggestions
> for how to scale up to >500 connections at once, and it's not that
> likely virtualization can fill any useful role in that context.

That rather depends on your virtualisation layer.  We haven't run large PG
databases on our zLinux/zVM machines, but we have Oracle DBs running
comparable connection numbers without any issues.

On Mon, Mar 29, 2010 at 4:50 PM, Ramiro Barreca <rbarreca@gmail.com> wrote:

> As this is a very heterogeneous platform (from Oracle 10g, SQL Server 2008,
> Firebird 2.1, Mysql 5, Postgre 8.4 up to Informix 5 and even COBOL apps) whe
> are evaluating Virtualization or just sharing a server among PostgreSQL and
> firebird or MySQL.
> As Virtualiaztion is almost discarded after reading many articles that
> locate this option in the "don'ts-list", we want to know your experience (if
> any) about installing a PostgreSQL and a Firebird server in the same Linux
> server.

Virtualisation is not a bad option per-se. Just don't expect the same
performance as if it was running on bare metal.

I have a VPS with Xen Hypervisor, 4 GB of RAM at the moment, running
PostgreSQL 8.4, MongoDB and MySQL. No problems at all. The setup
includes LVM, virtual disks are images on real disk. It's working just
fine, but it won't "hold" forever. It's a "cheap" solution at the
moment, affordable and performs very well. But I don't expect it to
perform as if there were no virtualisation layer.

> Is there a configuration option we need to consider to share this server?
> What is the recomended hardware for a PostgreSQL server that has to support
> between 500 and 1000 simultaneous user connections? Is there a whitepaper
> about this?

You definitely want to do connection pooling when possible, or try to
share tables among several database servers to spread the load. 500 up
to 1000 simultaneous connections are very resource consuming, no
matter which hardware you have. Of course it also depends on the type
of application your users are running...

--
Jaume Sabater
http://linuxsilo.net/

"Ubi sapientas ibi libertas"

Rodger Donaldson wrote:
> On Tue, March 30, 2010 06:09, Greg Smith wrote:
>
>> You answered your own question here.  Ramiro is looking for suggestions
>> for how to scale up to >500 connections at once, and it's not that
>> likely virtualization can fill any useful role in that context.
>>
>
> That rather depends on your virtualisation layer.  We haven't run large PG
> databases on our zLinux/zVM machines, but we have Oracle DBs running
> comparable connection numbers without any issues.
>

Connection scaling in Oracle doesn't have the same characteristics as
PostgreSQL, so you can't extrapolate from that.  My point was that the
connection target here would be aggressive and difficult to achieve even
without virtualization involved.  The virtualization sofware used will
impact the exact percentage of overhead involved, but you'd be hard
pressed to get this to work as hoped even if that number were 0--so
anything >0, even small, is increasing the odds of failure.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us

* Ramiro Barreca <rbarreca@gmail.com> wrote:

> As Virtualiaztion is almost discarded after reading many articles that
> locate this option in the "don'ts-list", we want to know your experience (if
> any) about installing a PostgreSQL and a Firebird server in the same Linux
> server.

Depends on the kind of virtualization. As you're on Linux, you could
use OpenVZ: it has similar characteristics as chroot - it just separates
namespaces (eg. network interfaces, pids, etc) and allows different
limits (eg. maxprocs, maxfiles, maxmem, ...) on per-VZ basis.

BTW: my current customer already consolidated multiple formerly
dedicated mysql servers (in mass hosting) onto one metal (now we're
building a system which allows moving individual databases around
in the cluster w/o being visible to the customer ...). The same
should be easily possible w/ PostgreSQL and other RDBM'es.


cu
--
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
     http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
    http://patches.metux.de/
---------------------------------------------------------------------