Thread: Yet another postgres scaling question (use on AMD Opteron)

Yet another postgres scaling question (use on AMD Opteron)

From
"Fred Moyer"
Date:
Greetings,

I am looking for feedback on the AMD Opteron based platform as a server
for Postgres, and also comments.  The databases I am looking at will
likely have aggregate sums of indexes larger than the amount of memory on
a 32 bit machine, so I am looking at 64-bit platforms.

Does anyone here have experience with use of the Opteron platform?  There
was a post a while ago by someone asking if it would run on a Beowolf
cluster of opterons ( indicated it won't ) but I am interested in knowing
if anyone has actually implemented it on an Opteron machine.  It should
work since Opteron runs linux, but knowing that it _is_ in production
somewhere would give me more leverage to get an implementation plan
approved.  I know that many here have used Sun and IBM to scale postgres
but looking at the cost, Opteron is the most attractive right now.

The database sizes I am looking at are in the realm of 5-50 gigabytes.  So
far I have been able to achieve acceptable speeds on a 2 gigabyte database
with a 32 bit machine, but as the data size grows an argument has been
introduced that going with MySQL would allow us to scale with smaller
machines (hold the flames please :).  I have optimized and tuned the
hardware and database according to the fine advice given on these lists
(especially according to the kernel cache/shared buffer cache balancing
threads).

Comments and suggestions greatly appreciated.

Thanks

Fred Moyer



Re: Yet another postgres scaling question (use on AMD Opteron)

From
Andrew Sullivan
Date:
On Sat, May 31, 2003 at 01:05:54PM -0700, Fred Moyer wrote:

> with a 32 bit machine, but as the data size grows an argument has been
> introduced that going with MySQL would allow us to scale with smaller
> machines (hold the flames please :).  I have optimized and tuned the

I haven't used Opterons, but I'm curious how that argument works.
I've never seen any evidence that MySQL is better with large
databases than PostgreSQL; am I just out of touch?

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: Yet another postgres scaling question (use on AMD Opteron)

From
"Fred Moyer"
Date:
I don't think you're out of touch - I think the people I'm trying to
convince are.  The crux of the problem I am dealing with is putting forth
a robust argument for scaling PostgreSQL with 64 bit machines for
increasing database sizes - Opteron was a natural choice because it is an
inexpensive 64-bit machine.  The MySQL argument was put forth by
non-technical people who are have not thoroughly evaluated the differences
between the two and have been swayed by marketing arguments.

I guess what I am looking for here are real-world scenarios where
PostgreSQL has been scaled using 64-bit.  I as the technical person know
the performance and scalability differences between running large datasets
on 64 bit versus 32 bit machines and can put forth technical reasons for
scaling Postgres as such - real world examples of it would validate my
arguments.


> On Sat, May 31, 2003 at 01:05:54PM -0700, Fred Moyer wrote:
>
>> with a 32 bit machine, but as the data size grows an argument has been
>> introduced that going with MySQL would allow us to scale with smaller
>> machines (hold the flames please :).  I have optimized and tuned the
>
> I haven't used Opterons, but I'm curious how that argument works.  I've
> never seen any evidence that MySQL is better with large
> databases than PostgreSQL; am I just out of touch?
>
> A
>
> --
> ----
> Andrew Sullivan                         204-4141 Yonge Street
> Liberty RMS                           Toronto, Ontario Canada
> <andrew@libertyrms.info>                              M2P 2A8
>                                          +1 416 646 3304 x110
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org


Fred Moyer
Digital Campaigns, Inc.




Re: Yet another postgres scaling question (use on AMD Opteron)

From
Douglas Trainor
Date:
The width of the memory bus is likely to be larger with a 64-bit machine
than a 32-bit machine, and that is a big factor for database applications.
But peak memory bandwidth numbers published by manufacturers are not
as useful as actual measured benchmarks for your type of problem.

PostgreSQL is great for scientific databases.  In the past, many database
systems were good at simple business applications, but almost none were
good for scientific applications.

    douglas

Fred Moyer wrote:

>I don't think you're out of touch - I think the people I'm trying to
>convince are.  The crux of the problem I am dealing with is putting forth
>a robust argument for scaling PostgreSQL with 64 bit machines for
>increasing database sizes - Opteron was a natural choice because it is an
>inexpensive 64-bit machine.  The MySQL argument was put forth by
>non-technical people who are have not thoroughly evaluated the differences
>between the two and have been swayed by marketing arguments.
>
>I guess what I am looking for here are real-world scenarios where
>PostgreSQL has been scaled using 64-bit.  I as the technical person know
>the performance and scalability differences between running large datasets
>on 64 bit versus 32 bit machines and can put forth technical reasons for
>scaling Postgres as such - real world examples of it would validate my
>arguments.
>
>[...]
>




Re: Yet another postgres scaling question (use on AMD Opteron)

From
Hans Spaans
Date:
On Sat, May 31, 2003 at 02:09:45PM -0700, Fred Moyer wrote:
> I don't think you're out of touch - I think the people I'm trying to
> convince are.  The crux of the problem I am dealing with is putting forth
> a robust argument for scaling PostgreSQL with 64 bit machines for
> increasing database sizes - Opteron was a natural choice because it is an
> inexpensive 64-bit machine.  The MySQL argument was put forth by
> non-technical people who are have not thoroughly evaluated the differences
> between the two and have been swayed by marketing arguments.
>
> I guess what I am looking for here are real-world scenarios where
> PostgreSQL has been scaled using 64-bit.  I as the technical person know
> the performance and scalability differences between running large datasets
> on 64 bit versus 32 bit machines and can put forth technical reasons for
> scaling Postgres as such - real world examples of it would validate my
> arguments.

You're mixing some theories. Because a machine is 64 bits doesn't mean
is just runs faster then 32 bits. Go figure why userland apps are
almost only 32 bits, even when running on 64 bits hardware. The
debian-sparc mailinglists has some good threads about. Also lkml had
some about increasing the size of the counter for the time from 32 to
64 bits and the impact.

Also 32 or 64 bits, it doesn't really matter to the max size of the
database and I must say Postgresql has a real nice way of fixing that
as far I have seen. And I don't really see this as an issue.

The only real pro about 64 bits is that the registers are bigger, but
then some processors just have more registers. So don't see 64 bits as
the holy grail for all the problems you have. And if you want to go 64
bits, go for a processor that is 64 bits for many years and has earned
his stars and strips IMHO.

--
Hans

Disk I/O Monitoring

From
Kris Kiger
Date:
Hi all, I'm looking for a good way to monitor disk I/O on a linux
system.  Specifically, I would like to know when postgres is making
page-ins and page-outs.  If anyone knows of a good utility, please let
me know!  Thank you for the help

Kris




Re: Disk I/O Monitoring

From
postgres timeless
Date:
On Fri, 2003-06-06 at 09:42, Kris Kiger wrote:
> I'm looking for a good way to monitor disk I/O on a linux
> system.  Specifically, I would like to know when postgres is making
> page-ins and page-outs.  If anyone knows of a good utility, please let
> me know!  Thank you for the help

vmstat.

--
Tim Ellis
Senior Database Architect and author, tedia2sql (http://tedia2sql.tigris.org)
If this helped you, http://svcs.affero.net/rm.php?r=philovivero



Re: Disk I/O Monitoring

From
"Mendola Gaetano"
Date:
wrote "Kris Kiger" <kris@musicrebellion.com>
> Hi all, I'm looking for a good way to monitor disk I/O on a linux
> system.  Specifically, I would like to know when postgres is making
> page-ins and page-outs.  If anyone knows of a good utility, please let
> me know!  Thank you for the help

try  iostat and vmstat

Regards
Gaetano Mendola


Re: Disk I/O Monitoring

From
Michael A Nachbaur
Date:
On Friday 06 June 2003 09:42 am, Kris Kiger wrote:
> Hi all, I'm looking for a good way to monitor disk I/O on a linux
> system.  Specifically, I would like to know when postgres is making
> page-ins and page-outs.  If anyone knows of a good utility, please let
> me know!  Thank you for the help

If you're on Linux, try KSysGuard, either the stand-alone app or the Kicker
applet.

--
Michael A Nachbaur <mike@nachbaur.com>