Thread: Yet another postgres scaling question (use on AMD Opteron)
Greetings, I am looking for feedback on the AMD Opteron based platform as a server for Postgres, and also comments. The databases I am looking at will likely have aggregate sums of indexes larger than the amount of memory on a 32 bit machine, so I am looking at 64-bit platforms. Does anyone here have experience with use of the Opteron platform? There was a post a while ago by someone asking if it would run on a Beowolf cluster of opterons ( indicated it won't ) but I am interested in knowing if anyone has actually implemented it on an Opteron machine. It should work since Opteron runs linux, but knowing that it _is_ in production somewhere would give me more leverage to get an implementation plan approved. I know that many here have used Sun and IBM to scale postgres but looking at the cost, Opteron is the most attractive right now. The database sizes I am looking at are in the realm of 5-50 gigabytes. So far I have been able to achieve acceptable speeds on a 2 gigabyte database with a 32 bit machine, but as the data size grows an argument has been introduced that going with MySQL would allow us to scale with smaller machines (hold the flames please :). I have optimized and tuned the hardware and database according to the fine advice given on these lists (especially according to the kernel cache/shared buffer cache balancing threads). Comments and suggestions greatly appreciated. Thanks Fred Moyer
On Sat, May 31, 2003 at 01:05:54PM -0700, Fred Moyer wrote:
> with a 32 bit machine, but as the data size grows an argument has been
> introduced that going with MySQL would allow us to scale with smaller
> machines (hold the flames please :). I have optimized and tuned the
I haven't used Opterons, but I'm curious how that argument works.
I've never seen any evidence that MySQL is better with large
databases than PostgreSQL; am I just out of touch?
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
I don't think you're out of touch - I think the people I'm trying to convince are. The crux of the problem I am dealing with is putting forth a robust argument for scaling PostgreSQL with 64 bit machines for increasing database sizes - Opteron was a natural choice because it is an inexpensive 64-bit machine. The MySQL argument was put forth by non-technical people who are have not thoroughly evaluated the differences between the two and have been swayed by marketing arguments. I guess what I am looking for here are real-world scenarios where PostgreSQL has been scaled using 64-bit. I as the technical person know the performance and scalability differences between running large datasets on 64 bit versus 32 bit machines and can put forth technical reasons for scaling Postgres as such - real world examples of it would validate my arguments. > On Sat, May 31, 2003 at 01:05:54PM -0700, Fred Moyer wrote: > >> with a 32 bit machine, but as the data size grows an argument has been >> introduced that going with MySQL would allow us to scale with smaller >> machines (hold the flames please :). I have optimized and tuned the > > I haven't used Opterons, but I'm curious how that argument works. I've > never seen any evidence that MySQL is better with large > databases than PostgreSQL; am I just out of touch? > > A > > -- > ---- > Andrew Sullivan 204-4141 Yonge Street > Liberty RMS Toronto, Ontario Canada > <andrew@libertyrms.info> M2P 2A8 > +1 416 646 3304 x110 > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org Fred Moyer Digital Campaigns, Inc.
The width of the memory bus is likely to be larger with a 64-bit machine
than a 32-bit machine, and that is a big factor for database applications.
But peak memory bandwidth numbers published by manufacturers are not
as useful as actual measured benchmarks for your type of problem.
PostgreSQL is great for scientific databases. In the past, many database
systems were good at simple business applications, but almost none were
good for scientific applications.
douglas
Fred Moyer wrote:
>I don't think you're out of touch - I think the people I'm trying to
>convince are. The crux of the problem I am dealing with is putting forth
>a robust argument for scaling PostgreSQL with 64 bit machines for
>increasing database sizes - Opteron was a natural choice because it is an
>inexpensive 64-bit machine. The MySQL argument was put forth by
>non-technical people who are have not thoroughly evaluated the differences
>between the two and have been swayed by marketing arguments.
>
>I guess what I am looking for here are real-world scenarios where
>PostgreSQL has been scaled using 64-bit. I as the technical person know
>the performance and scalability differences between running large datasets
>on 64 bit versus 32 bit machines and can put forth technical reasons for
>scaling Postgres as such - real world examples of it would validate my
>arguments.
>
>[...]
>
On Sat, May 31, 2003 at 02:09:45PM -0700, Fred Moyer wrote: > I don't think you're out of touch - I think the people I'm trying to > convince are. The crux of the problem I am dealing with is putting forth > a robust argument for scaling PostgreSQL with 64 bit machines for > increasing database sizes - Opteron was a natural choice because it is an > inexpensive 64-bit machine. The MySQL argument was put forth by > non-technical people who are have not thoroughly evaluated the differences > between the two and have been swayed by marketing arguments. > > I guess what I am looking for here are real-world scenarios where > PostgreSQL has been scaled using 64-bit. I as the technical person know > the performance and scalability differences between running large datasets > on 64 bit versus 32 bit machines and can put forth technical reasons for > scaling Postgres as such - real world examples of it would validate my > arguments. You're mixing some theories. Because a machine is 64 bits doesn't mean is just runs faster then 32 bits. Go figure why userland apps are almost only 32 bits, even when running on 64 bits hardware. The debian-sparc mailinglists has some good threads about. Also lkml had some about increasing the size of the counter for the time from 32 to 64 bits and the impact. Also 32 or 64 bits, it doesn't really matter to the max size of the database and I must say Postgresql has a real nice way of fixing that as far I have seen. And I don't really see this as an issue. The only real pro about 64 bits is that the registers are bigger, but then some processors just have more registers. So don't see 64 bits as the holy grail for all the problems you have. And if you want to go 64 bits, go for a processor that is 64 bits for many years and has earned his stars and strips IMHO. -- Hans
Hi all, I'm looking for a good way to monitor disk I/O on a linux system. Specifically, I would like to know when postgres is making page-ins and page-outs. If anyone knows of a good utility, please let me know! Thank you for the help Kris
On Fri, 2003-06-06 at 09:42, Kris Kiger wrote: > I'm looking for a good way to monitor disk I/O on a linux > system. Specifically, I would like to know when postgres is making > page-ins and page-outs. If anyone knows of a good utility, please let > me know! Thank you for the help vmstat. -- Tim Ellis Senior Database Architect and author, tedia2sql (http://tedia2sql.tigris.org) If this helped you, http://svcs.affero.net/rm.php?r=philovivero
wrote "Kris Kiger" <kris@musicrebellion.com> > Hi all, I'm looking for a good way to monitor disk I/O on a linux > system. Specifically, I would like to know when postgres is making > page-ins and page-outs. If anyone knows of a good utility, please let > me know! Thank you for the help try iostat and vmstat Regards Gaetano Mendola
On Friday 06 June 2003 09:42 am, Kris Kiger wrote: > Hi all, I'm looking for a good way to monitor disk I/O on a linux > system. Specifically, I would like to know when postgres is making > page-ins and page-outs. If anyone knows of a good utility, please let > me know! Thank you for the help If you're on Linux, try KSysGuard, either the stand-alone app or the Kicker applet. -- Michael A Nachbaur <mike@nachbaur.com>