Thread: Hardware selection
As mentioned previously I have a large text database with upwards of 40GB of data and 8 million tuples. The time has come to buy some real hardware for it. Having read around the subject online I see the general idea is to get as much memory and the fastest I/O possible. The buget for the server is approx 20K GBP / 30K US$ The data will be stored on an external raid, SCSI based 2.5TB with IDE disks. Configured as 1 large volume, RAID5. ( We already have this hardware) Question 1 - whats the best raid configuration for this device? Onto the server, I have currently priced up the following: Dell Poweredge 6650 16GB RAM Quad intel hyperthreaded arch PERC3Q controller 36GB System HDD approx 20K UKP Question 2 - Comments and opinions on this as the postgresql server. Any comments on better alternatives or modification to this system would be very helpful! Question 3 - what would be the best way to distribute tables/swap/ system disk space? Hope I'm not asking too much... Thanks !
On Fri, 2003-07-25 at 07:42, psql-mail@freeuk.com wrote: > As mentioned previously I have a large text database with upwards of > 40GB of data and 8 million tuples. > > The time has come to buy some real hardware for it. > > Having read around the subject online I see the general idea is to get > as much memory and the fastest I/O possible. > > The buget for the server is approx 20K GBP / 30K US$ > > The data will be stored on an external raid, > SCSI based 2.5TB with IDE disks. Configured as 1 large volume, RAID5. ( > We already have this hardware) There's a SCSI-IDE bridge somewhere in there? 40GB of data, and 2.5TB disk space?? Even with RAID1+0, that's 1.25TB usable; 40GB (even after accounting for overhead and indexes) would get lost in 1250GB. Planning for growth, or also storing lots of non-DB related files (which is not a good idea)? > Question 1 - whats the best raid configuration for this device? What's your read-to-write ratio? Few complex multi-table queries, lots of simple queries, or a mixture? > Onto the server, I have currently priced up the following: > Dell Poweredge 6650 > 16GB RAM > Quad intel hyperthreaded arch > PERC3Q controller > 36GB System HDD > approx 20K UKP Because of the limits of the x86 architecture, with 16GB RAM, I'd really think about a 64 bit system. Opteron, Alpha, Power5, Sparc64, etc. You can put 16GB RAM in x86 systems, but 64 bit systems use it more efficiently. > Question 2 - Comments and opinions on this as the postgresql server. > Any comments on better alternatives or modification to this system > would be very helpful! > > Question 3 - what would be the best way to distribute tables/swap/ > system disk space? 16GB swap would fit nicely on the system disk. Hope it's on a separate channel from the DB... What OS are you using? > Hope I'm not asking too much... > Thanks ! -- +-----------------------------------------------------------------+ | Ron Johnson, Jr. Home: ron.l.johnson@cox.net | | Jefferson, LA USA | | | | "I'm not a vegetarian because I love animals, I'm a vegetarian | | because I hate vegetables!" | | unknown | +-----------------------------------------------------------------+
Ron thank you for your comments, sorry of the slow response - i actually replied to you on saturday but i think the list was having trouble again?! Your questions are answered below... > On Fri, 2003-07-25 at 07:42, psql-mail@freeuk.com wrote: > > As mentioned previously I have a large text database with upwards of > > > 40GB of data and 8 million tuples. > > > > The time has come to buy some real hardware for it. > > > > Having read around the subject online I see the general idea is to > get > > as much memory and the fastest I/O possible. > > > > The buget for the server is approx 20K GBP / 30K US$ > > > > The data will be stored on an external raid, > > SCSI based 2.5TB with IDE disks. Configured as 1 large volume, RAID5. > ( > > We already have this hardware) > > There's a SCSI-IDE bridge somewhere in there? The raid hardware takes care of it, IDE disks slot into the backplane, the RAID then SCSI attached to the server. If this hardware is percieved to be the bottleneck - and i think it might be, then whats the best thing to replace it with? (a few different specs/prices would be very helpful!) > 40GB of data, and 2.5TB disk space?? Even with RAID1+0, that's > 1.25TB usable; 40GB (even after accounting for overhead and indexes) > would get lost in 1250GB. Planning for growth, or also storing lots > of non-DB related files (which is not a good idea)? Data is currently 40GB, maybe a bit more, it will increase steadily in the future. There will be no non-database files on there. I am also planning to do a full text index on the data so that will take up alot of room. > > Question 1 - whats the best raid configuration for this device? > > What's your read-to-write ratio? Faily small bulk INSERTS every 30 mins. Continous querying within office hours. > Few complex multi-table queries, lots of simple queries, or > a mixture? Mostley queries of the form 'ILIKE "%chicken%"' across >= 30GB of data. This may change if full text indexing is sucessful but will still be required for phrases 'ILIKE "%chicken and chips%" There will also be lots of smaller queries going on constantly, but these are already quick enough on the old hardware. > > Onto the server, I have currently priced up the following: > > Dell Poweredge 6650 > > 16GB RAM > > Quad intel hyperthreaded arch > > PERC3Q controller > > 36GB System HDD > > approx 20K UKP > > Because of the limits of the x86 architecture, with 16GB RAM, I'd > really think about a 64 bit system. Opteron, Alpha, Power5, > Sparc64, etc. > > You can put 16GB RAM in x86 systems, but 64 bit systems use it > more efficiently. I was under the impression that both the O/S and the application ( postmaster) needed to be optimised to make best use of 64 bit arch. Is this impression wrong? If not, is postgresql optimised for 64bit arch? > > Question 2 - Comments and opinions on this as the postgresql server. > > > Any comments on better alternatives or modification to this system > > would be very helpful! > > > > Question 3 - what would be the best way to distribute tables/swap/ > > system disk space? > > 16GB swap would fit nicely on the system disk. Hope it's on a > separate channel from the DB... Will put the swap on the local disk and the data on the external RAID - different channels. > What OS are you using? I will be using Red hat advanced server 2.1 - thats not set in stone yet but RH is the distro that I am most familiar with. Thanks for you help so far! PS. Just in case theres an amazing system for 60K or less point it out in case i can get more funding! --
On Mon, 2003-07-28 at 11:28, psql-mail@freeuk.com wrote: > Ron thank you for your comments, sorry of the slow response - i > actually replied to you on saturday but i think the list was having > trouble again?! > > Your questions are answered below... > > > On Fri, 2003-07-25 at 07:42, psql-mail@freeuk.com wrote: > > > As mentioned previously I have a large text database with upwards > of > > > > > 40GB of data and 8 million tuples. > > > > > > The time has come to buy some real hardware for it. > > > > > > Having read around the subject online I see the general idea is to > > > get > > > as much memory and the fastest I/O possible. > > > > > > The buget for the server is approx 20K GBP / 30K US$ > > > > > > The data will be stored on an external raid, > > > SCSI based 2.5TB with IDE disks. Configured as 1 large volume, > RAID5. > > ( > > > We already have this hardware) > > > > There's a SCSI-IDE bridge somewhere in there? > > The raid hardware takes care of it, IDE disks slot into the backplane, > the RAID then SCSI attached to the server. > > If this hardware is percieved to be the bottleneck - and i think it > might be, then whats the best thing to replace it with? (a few > different specs/prices would be very helpful!) > > > 40GB of data, and 2.5TB disk space?? Even with RAID1+0, that's > > 1.25TB usable; 40GB (even after accounting for overhead and indexes) > > would get lost in 1250GB. Planning for growth, or also storing lots > > of non-DB related files (which is not a good idea)? > > Data is currently 40GB, maybe a bit more, it will increase steadily in > the future. There will be no non-database files on there. > > I am also planning to do a full text index on the data so that will > take up alot of room. Besides, if you can get the disk space, take it! Will these be 250GB drives? What kind of DB backup scheme do you have planned? > > > Question 1 - whats the best raid configuration for this device? > > > > What's your read-to-write ratio? > > Faily small bulk INSERTS every 30 mins. > Continous querying within office hours. > > > Few complex multi-table queries, lots of simple queries, or > > a mixture? > > Mostley queries of the form 'ILIKE "%chicken%"' across >= 30GB of data. > This may change if full text indexing is sucessful but will still be > required for phrases 'ILIKE "%chicken and chips%" > > There will also be lots of smaller queries going on constantly, but > these are already quick enough on the old hardware. > > > > Onto the server, I have currently priced up the following: > > > Dell Poweredge 6650 > > > 16GB RAM > > > Quad intel hyperthreaded arch > > > PERC3Q controller > > > 36GB System HDD > > > approx 20K UKP > > > > Because of the limits of the x86 architecture, with 16GB RAM, I'd > > really think about a 64 bit system. Opteron, Alpha, Power5, > > Sparc64, etc. > > > > You can put 16GB RAM in x86 systems, but 64 bit systems use it > > more efficiently. > > I was under the impression that both the O/S and the application ( > postmaster) needed to be optimised to make best use of 64 bit arch. > Is this impression wrong? If not, is postgresql optimised for 64bit > arch? PG mostly uses the OS' cache, so if it's a 64 bit OS, and PG is compiled for the target, things will be fine. SuSE has an Opteron distro: http://www.suse.co.uk/uk/business/products/server/sles/index.html http://www.suse.co.uk/en/business/products/server/sles/misc/sles8_amd64.pdf > > > Question 2 - Comments and opinions on this as the postgresql server. > > > > > > Any comments on better alternatives or modification to this system > > > > would be very helpful! > > > > > > Question 3 - what would be the best way to distribute tables/swap/ > > > system disk space? > > > > 16GB swap would fit nicely on the system disk. Hope it's on a > > separate channel from the DB... > > Will put the swap on the local disk and the data on the external RAID - > different channels. > > > What OS are you using? > I will be using Red hat advanced server 2.1 - thats not set in stone > yet but RH is the distro that I am most familiar with. > > Thanks for you help so far! > > PS. Just in case theres an amazing system for 60K or less point it out > in case i can get more funding! -- +-----------------------------------------------------------------+ | Ron Johnson, Jr. Home: ron.l.johnson@cox.net | | Jefferson, LA USA | | | | "I'm not a vegetarian because I love animals, I'm a vegetarian | | because I hate vegetables!" | | unknown | +-----------------------------------------------------------------+
On Fri, 25 Jul 2003 psql-mail@freeuk.com wrote: > The data will be stored on an external raid, > SCSI based 2.5TB with IDE disks. Configured as 1 large volume, RAID5. ( > We already have this hardware) How come you did not go with SCSI disks? Specially 15K ones. Performance will be much better with RAID 1+0/0+1 than with RAID 5. > Dell Poweredge 6650 > 16GB RAM > Quad intel hyperthreaded arch As others mentioned you may be better off going with a 64bit architecture. I have no experience with that, but you consider a SUN machine or a HP Alpha. > PERC3Q controller > 36GB System HDD > approx 20K UKP My advice would be get just a 2 CPU machine, get a better disk subsystem and get only 8GB of RAM (or get 64bit and 16GB RAM). > Any comments on better alternatives or modification to this system > would be very helpful! You REALLY need to think out your DB structures and your index schemes. That could mean all the difference. A good DB organization will help you probably more than just simply throwing hardware at the problem. > Question 3 - what would be the best way to distribute tables/swap/ > system disk space? Throw it all at your RAID 1+0 device.