Thread: Postgres over Linux NBD or NFS
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We're looking for an alternative to fiber-channel disk arrays for mass storage. One of the ideas that we're exploring would involve having the cluster on an NFS mounted filesystem. Another technology we're looking at is the Linux NBD (Network Block Device). Has anyone had any experience with running postgres over either of these technologies? What issues do we need to know about / pay attention to? - -- Andrew Hammond 416-673-4138 ahammond@ca.afilias.info Database Administrator, Afilias Canada Corp. CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFA1yK6gfzn5SevSpoRAtYAAKCrghfAKV5kVuiTd/2TOwEbr4Q7hACgr3rT mEvFi8AOHX9I43T45fH1e0U= =1Cs9 -----END PGP SIGNATURE-----
Attachment
Andrew Hammond <ahammond@ca.afilias.info> writes:
> We're looking for an alternative to fiber-channel disk arrays for mass
> storage. One of the ideas that we're exploring would involve having the
> cluster on an NFS mounted filesystem. Another technology we're looking
> at is the Linux NBD (Network Block Device).
There are a lot of horror stories concerning running databases (not only
Postgres) over NFS. I wouldn't recommend it. Dunno anything about NBD
though.
regards, tom lane
On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > We're looking for an alternative to fiber-channel disk arrays for mass > storage. One of the ideas that we're exploring would involve having the > cluster on an NFS mounted filesystem. Another technology we're looking > at is the Linux NBD (Network Block Device). > No idea about NBDs, but its generally accepted that running over NFS would significantly decrease reliability and performance, i.e. it would be a Bad Move (tm). Not sure what you think to gain. I sure wouldn't trust NFS with a production database. What exactly are you trying to gain, avoid, or do? > Has anyone had any experience with running postgres over either of > these > technologies? What issues do we need to know about / pay attention to? > > - -- > Andrew Hammond 416-673-4138 ahammond@ca.afilias.info > Database Administrator, Afilias Canada Corp. > CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.4 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > > iD8DBQFA1yK6gfzn5SevSpoRAtYAAKCrghfAKV5kVuiTd/2TOwEbr4Q7hACgr3rT > mEvFi8AOHX9I43T45fH1e0U= > =1Cs9 > -----END PGP SIGNATURE----- > <ahammond.vcf> > ---------------------------(end of > broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to > majordomo@postgresql.org) > -------------------- Andrew Rawnsley President The Ravensfield Digital Resource Group, Ltd. (740) 587-0114 www.ravensfield.com
On Mon, 2004-06-21 at 20:46, Andrew Rawnsley wrote: > On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > We're looking for an alternative to fiber-channel disk arrays for mass > > storage. One of the ideas that we're exploring would involve having the > > cluster on an NFS mounted filesystem. Another technology we're looking > > at is the Linux NBD (Network Block Device). > > > > No idea about NBDs, but its generally accepted that running over NFS > would significantly > decrease reliability and performance, i.e. it would be a Bad Move (tm). > Not sure what you > think to gain. I sure wouldn't trust NFS with a production database. > > What exactly are you trying to gain, avoid, or do? I've gotten good performance over NFS using switched 100, then later gigabit. But I wouldn't trust it for diddly.
In an attempt to throw the authorities off his trail, ronz@ravensfield.com (Andrew Rawnsley) transmitted:
> On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
>> We're looking for an alternative to fiber-channel disk arrays for mass
>> storage. One of the ideas that we're exploring would involve having the
>> cluster on an NFS mounted filesystem. Another technology we're looking
>> at is the Linux NBD (Network Block Device).
>
> No idea about NBDs, but its generally accepted that running over NFS
> would significantly decrease reliability and performance, i.e. it
> would be a Bad Move (tm). Not sure what you think to gain. I sure
> wouldn't trust NFS with a production database.
>
> What exactly are you trying to gain, avoid, or do?
The point of the exercise is to try to come up with something that is
a near-substitute for a SAN.
With a SAN, you have a box with a whole lot of disk in it, and then
your database servers connect to that box, typically via something
like fibrechannel.
One of the goals is for this to allow trying out Opterons at low risk.
Should performance turn out to suck or there be some other
disqualification, it's simple to hook the disk up to something else
instead.
The other goal is to be able to stick LOTS of disk into one box, and
dole it out to multiple servers. It's more expensive to set up and
manage 3 RAID arrays than it is to set up and manage just 1, because
you have to manage 3 sets of disk hardware rather than 1.
But I'm getting convinced that the attempt to get this clever about it
is counterproductive unless you have outrageous amounts of money to
throw at it.
- NFS may well be acceptable if you buy into something with potent FS
semantics, as with NetApp boxes. But they're REALLY expensive.
- FibreChannel offers interesting options in conjunction with a fairly
smart SAN box and Veritas, where you could have 5TB of storage in
one box, and then assign 2TB apiece to two servers, and the other
1TB to a third. But the pricing premium again leaps out at ya.
The "poor man's approach" involves trying to fake this by building a
"disk box" running Linux that exports the storage either as a
filesystem (using NFS) or as disk blocks (NBD). NFS clearly doesn't
provide the filesystem semantics needed to get decent reliability;
with NBD, it's not clear what happens :-(.
Barring that, it means building a separate RAID array for each server,
and living with the limitation that a full set of disk hardware has to
be devoted to each DB server.
--
wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
http://www3.sympatico.ca/cbbrowne/
Rules of the Evil Overlord #46. "If an advisor says to me "My liege,
he is but one man. What can one man possibly do?", I will reply
"This." and kill the advisor." <http://www.eviloverlord.com/>
There are some less expensive baby-SAN options coming out now - Dell
has an
rebranded EMC baby SAN (which of course doesn't work with any other EMC
system...) that starts at about $6000 or so. Just read the announcement
- don't know
anything else. While there have been some reports of undewhelming
performance for
database applications, the Apple XRaid has a sweet price point,
particularly if you're
in an industry that they want some exposure in (we're in financial
services, they almost
gave it to us...$7000 for 2TB, batteries, accessory kits, etc), and a
decent feature
set. It works with non-Apple stuff..
The baby-SANs don't necessarily do many of the things that you can get
out of a full-blown
EMC/NetApp rig, but then again, you're not paying for it either.
There are a lot of lower-cost storage options popping up now, as
IDE/SATA disks arrays
proliferate. You can get external RAID boxes that talk SCSI or fiber
with IDE disks for
dirt these days. Small, too. Portable. I'm see little need to buy
massive boxes with
internal storage arrays anymore.
On Jun 22, 2004, at 7:50 AM, Christopher Browne wrote:
> In an attempt to throw the authorities off his trail,
> ronz@ravensfield.com (Andrew Rawnsley) transmitted:
>> On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote:
>>> We're looking for an alternative to fiber-channel disk arrays for
>>> mass
>>> storage. One of the ideas that we're exploring would involve having
>>> the
>>> cluster on an NFS mounted filesystem. Another technology we're
>>> looking
>>> at is the Linux NBD (Network Block Device).
>>
>> No idea about NBDs, but its generally accepted that running over NFS
>> would significantly decrease reliability and performance, i.e. it
>> would be a Bad Move (tm). Not sure what you think to gain. I sure
>> wouldn't trust NFS with a production database.
>>
>> What exactly are you trying to gain, avoid, or do?
>
> The point of the exercise is to try to come up with something that is
> a near-substitute for a SAN.
>
> With a SAN, you have a box with a whole lot of disk in it, and then
> your database servers connect to that box, typically via something
> like fibrechannel.
>
> One of the goals is for this to allow trying out Opterons at low risk.
> Should performance turn out to suck or there be some other
> disqualification, it's simple to hook the disk up to something else
> instead.
>
> The other goal is to be able to stick LOTS of disk into one box, and
> dole it out to multiple servers. It's more expensive to set up and
> manage 3 RAID arrays than it is to set up and manage just 1, because
> you have to manage 3 sets of disk hardware rather than 1.
>
> But I'm getting convinced that the attempt to get this clever about it
> is counterproductive unless you have outrageous amounts of money to
> throw at it.
>
> - NFS may well be acceptable if you buy into something with potent FS
> semantics, as with NetApp boxes. But they're REALLY expensive.
>
> - FibreChannel offers interesting options in conjunction with a fairly
> smart SAN box and Veritas, where you could have 5TB of storage in
> one box, and then assign 2TB apiece to two servers, and the other
> 1TB to a third. But the pricing premium again leaps out at ya.
>
> The "poor man's approach" involves trying to fake this by building a
> "disk box" running Linux that exports the storage either as a
> filesystem (using NFS) or as disk blocks (NBD). NFS clearly doesn't
> provide the filesystem semantics needed to get decent reliability;
> with NBD, it's not clear what happens :-(.
>
> Barring that, it means building a separate RAID array for each server,
> and living with the limitation that a full set of disk hardware has to
> be devoted to each DB server.
> --
> wm(X,Y):-write(X),write('@'),write(Y). wm('cbbrowne','acm.org').
> http://www3.sympatico.ca/cbbrowne/
> Rules of the Evil Overlord #46. "If an advisor says to me "My liege,
> he is but one man. What can one man possibly do?", I will reply
> "This." and kill the advisor." <http://www.eviloverlord.com/>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>
--------------------
Andrew Rawnsley
President
The Ravensfield Digital Resource Group, Ltd.
(740) 587-0114
www.ravensfield.com
How about iSCSI? This is exactly what it's for - presenting a bunch of remote SCSI hardware as if it were local. There are several reference implementations on SourceForge from Intel, Cisco & others. I've never tried it myself, but I would if I had the need. And let's face it there are some very big players selling very pricey kit that uses it, so you should have pretty high confidence that the fundamentals are strong. M > The other goal is to be able to stick LOTS of disk into one > box, and dole it out to multiple servers. It's more > expensive to set up and manage 3 RAID arrays than it is to > set up and manage just 1, because you have to manage 3 sets > of disk hardware rather than 1. [snip] > The "poor man's approach" involves trying to fake this by > building a "disk box" running Linux that exports the storage > either as a filesystem (using NFS) or as disk blocks (NBD). > NFS clearly doesn't provide the filesystem semantics needed > to get decent reliability; with NBD, it's not clear what happens :-(.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 |> What exactly are you trying to gain, avoid, or do? Gain: seperate database storage from processing. This lets me move clusters from one server to another easily. Just stop the postgres instance on server A and unmount it's filesystem. Then mount it on server B and start postgres instance on server B. It gives me some fail-over capability as well as scalability and a lot of flexibility in balancing load over multiple servers. Avoid: paying for brutally expensive FC gear. - -- Andrew Hammond 416-673-4138 ahammond@ca.afilias.info Database Administrator, Afilias Canada Corp. CB83 2838 4B67 D40F D086 3568 81FC E7E5 27AF 4A9A -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFA2Djagfzn5SevSpoRAj+bAKDFFgrhX+G1gkZRrydow3j/j35VaACbBN3Y C/0nWmqcwo/UlqvYpng06Ks= =k2vg -----END PGP SIGNATURE-----
Attachment
Anselmo bom dia! Não é custoso monstar um Cluster (Storage caseiro) em PostgreSQL, Está em discussão no forum da PostgreSQL a possíbilidade de usar o NFS (Network file system) ou o NBD (Network Block Device), ambos consistem em "Mapear" a partição de dados do PostgreSQL em uma OUTRA máquina com PostgreSQL a fim de que os as duas máquinas trabalhem com a mesma base de dados. Carlos Eduardo Smanioto Infra Estrutura - Servidores e Segurança Planae - Tecnologia da Informação Fone/Fax +55 14 3224-3066 Ramal 207 www.planae.com.br ----- Original Message ----- From: "Scott Marlowe" <smarlowe@qwest.net> To: "Andrew Rawnsley" <ronz@ravensfield.com> Cc: "Andrew Hammond" <ahammond@ca.afilias.info>; <pgsql-performance@postgresql.org> Sent: Tuesday, June 22, 2004 12:46 AM Subject: Re: [PERFORM] Postgres over Linux NBD or NFS > On Mon, 2004-06-21 at 20:46, Andrew Rawnsley wrote: > > On Jun 21, 2004, at 2:02 PM, Andrew Hammond wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA1 > > > > > > We're looking for an alternative to fiber-channel disk arrays for mass > > > storage. One of the ideas that we're exploring would involve having the > > > cluster on an NFS mounted filesystem. Another technology we're looking > > > at is the Linux NBD (Network Block Device). > > > > > > > No idea about NBDs, but its generally accepted that running over NFS > > would significantly > > decrease reliability and performance, i.e. it would be a Bad Move (tm). > > Not sure what you > > think to gain. I sure wouldn't trust NFS with a production database. > > > > What exactly are you trying to gain, avoid, or do? > > I've gotten good performance over NFS using switched 100, then later > gigabit. But I wouldn't trust it for diddly. > > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings
I just got the BigAdmin newsletter from Sun today... interestingly enough it had a link to an article described as: > Database Performance with NAS: Optimizing Oracle on NFS > This paper discusses the operation of relational databases with network > attached storage (NAS). Management implications and performance > expectations for databases using NAS are presented. The link points to: http://www.sun.com/bigadmin/content/nas/ I read just enough to see if it is relevant. Here is the first part of the summary: > IT departments are increasingly utilizing Network Attached Storage (NAS) > and the Network File System (NFS) to meet the storage needs of mission- > critical relational databases. Reasons for this adoption include improved > storage virtualization, ease of storage deployment, decreased complexity, > and decreased total cost of ownership. This paper directly examines the > performance of databases with NAS. In laboratory tests comparing NFS with > local storage, NFS is shown capable of sustaining the same workload level > as local storage. Under similar workload conditions, NFS does consume an > increased number of CPU cycles; however, the proven benefits of NFS and > NAS outweigh this penalty in most production environments. Matthew Nuzum | ISPs: Make $200 - $5,000 per referral by www.followers.net | recomending Elite CMS to your customers! matt@followers.net | http://www.followers.net/isp