Thread: DAFS?
Hiya
We use NetApp filers for network attached storage, these support
the DAFS protocol. Is there any support or plan for support of this in
PostgreSQL? Would be VERY handy if it was supported.
DAFS info can be obtained at:
http://www.dafscollaborative.org
-----
Graeme Hinchliffe BSc (Hons) _
Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( )
- against HTML email X
VIA NET.WORKS & vCards / \
http://arc.pasp.de/
Tel +44-1925-484065
Fax +44-1925-484055
ICQ : 3842605 (name : Link)
www.vianetworks.co.uk
I know some folks are using NetApp and haven't had any problems. We don't officially support it because we discourage NFS because the file semantics are not the same as Unix file semantics. --------------------------------------------------------------------------- ghinchliffe@vianetworks.co.uk wrote: > Hiya > We use NetApp filers for network attached storage, these support > the DAFS protocol. Is there any support or plan for support of this in > PostgreSQL? Would be VERY handy if it was supported. > > DAFS info can be obtained at: > > http://www.dafscollaborative.org > > ----- > Graeme Hinchliffe BSc (Hons) _ > Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) > - against HTML email X > VIA NET.WORKS & vCards / \ > http://arc.pasp.de/ > Tel +44-1925-484065 > Fax +44-1925-484055 > > ICQ : 3842605 (name : Link) > > www.vianetworks.co.uk > > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
Any feature (or current) support of raw devices. With iSCSI on the move, that would be a good thing. Bruce Momjian wrote: >I know some folks are using NetApp and haven't had any problems. We >don't officially support it because we discourage NFS because the file >semantics are not the same as Unix file semantics. > >--------------------------------------------------------------------------- > >ghinchliffe@vianetworks.co.uk wrote: > > >>Hiya >> We use NetApp filers for network attached storage, these support >>the DAFS protocol. Is there any support or plan for support of this in >>PostgreSQL? Would be VERY handy if it was supported. >> >>DAFS info can be obtained at: >> >>http://www.dafscollaborative.org >> >>----- >>Graeme Hinchliffe BSc (Hons) _ >>Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) >> - against HTML email X >>VIA NET.WORKS & vCards / \ >> http://arc.pasp.de/ >>Tel +44-1925-484065 >>Fax +44-1925-484055 >> >>ICQ : 3842605 (name : Link) >> >>www.vianetworks.co.uk >> >> >>---------------------------(end of broadcast)--------------------------- >>TIP 4: Don't 'kill -9' the postmaster >> >> >> > > >
No, we feel raw devices would be only a small win with a major pain to implement. --------------------------------------------------------------------------- Medi Montaseri wrote: > Any feature (or current) support of raw devices. With iSCSI on the move, > that would be > a good thing. > > Bruce Momjian wrote: > > >I know some folks are using NetApp and haven't had any problems. We > >don't officially support it because we discourage NFS because the file > >semantics are not the same as Unix file semantics. > > > >--------------------------------------------------------------------------- > > > >ghinchliffe@vianetworks.co.uk wrote: > > > > > >>Hiya > >> We use NetApp filers for network attached storage, these support > >>the DAFS protocol. Is there any support or plan for support of this in > >>PostgreSQL? Would be VERY handy if it was supported. > >> > >>DAFS info can be obtained at: > >> > >>http://www.dafscollaborative.org > >> > >>----- > >>Graeme Hinchliffe BSc (Hons) _ > >>Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) > >> - against HTML email X > >>VIA NET.WORKS & vCards / \ > >> http://arc.pasp.de/ > >>Tel +44-1925-484065 > >>Fax +44-1925-484055 > >> > >>ICQ : 3842605 (name : Link) > >> > >>www.vianetworks.co.uk > >> > >> > >>---------------------------(end of broadcast)--------------------------- > >>TIP 4: Don't 'kill -9' the postmaster > >> > >> > >> > > > > > > > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
FYI, some of my kernel engineer friends are totally scared of File Systems on top of block devices, particularly the buffer cache....maybe they know something....I was always under the impression that DB engine writers don't like to deal with Filesystems as a file could consists of spread blocks (even zero padded ones) and the only way to be sure that you do have contingues blocks is to go after the raw device itself. Anyways, thanks for the reply.... Bruce Momjian wrote: >No, we feel raw devices would be only a small win with a major pain to implement. > >--------------------------------------------------------------------------- > >Medi Montaseri wrote: > > >>Any feature (or current) support of raw devices. With iSCSI on the move, >>that would be >>a good thing. >> >>Bruce Momjian wrote: >> >> >> >>>I know some folks are using NetApp and haven't had any problems. We >>>don't officially support it because we discourage NFS because the file >>>semantics are not the same as Unix file semantics. >>> >>>--------------------------------------------------------------------------- >>> >>>ghinchliffe@vianetworks.co.uk wrote: >>> >>> >>> >>> >>>>Hiya >>>> We use NetApp filers for network attached storage, these support >>>>the DAFS protocol. Is there any support or plan for support of this in >>>>PostgreSQL? Would be VERY handy if it was supported. >>>> >>>>DAFS info can be obtained at: >>>> >>>>http://www.dafscollaborative.org >>>> >>>>----- >>>>Graeme Hinchliffe BSc (Hons) _ >>>>Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) >>>> - against HTML email X >>>>VIA NET.WORKS & vCards / \ >>>> http://arc.pasp.de/ >>>>Tel +44-1925-484065 >>>>Fax +44-1925-484055 >>>> >>>>ICQ : 3842605 (name : Link) >>>> >>>>www.vianetworks.co.uk >>>> >>>> >>>>---------------------------(end of broadcast)--------------------------- >>>>TIP 4: Don't 'kill -9' the postmaster >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >> >> >>---------------------------(end of broadcast)--------------------------- >>TIP 5: Have you checked our extensive FAQ? >> >>http://www.postgresql.org/users-lounge/docs/faq.html >> >> >> > > >
but dafs isn't nfs. we were trialing netapp for storage but are going back to local disks as the db ran VERY slowly in comparison. dafs should accelerate things from what i have read. graeme On Tue, 29 Oct 2002, Bruce Momjian wrote: > > I know some folks are using NetApp and haven't had any problems. We > don't officially support it because we discourage NFS because the file > semantics are not the same as Unix file semantics. > > --------------------------------------------------------------------------- > > ghinchliffe@vianetworks.co.uk wrote: > > Hiya > > We use NetApp filers for network attached storage, these support > > the DAFS protocol. Is there any support or plan for support of this in > > PostgreSQL? Would be VERY handy if it was supported. > > > > DAFS info can be obtained at: > > > > http://www.dafscollaborative.org > > > > ----- > > Graeme Hinchliffe BSc (Hons) _ > > Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) > > - against HTML email X > > VIA NET.WORKS & vCards / \ > > http://arc.pasp.de/ > > Tel +44-1925-484065 > > Fax +44-1925-484055 > > > > ICQ : 3842605 (name : Link) > > > > www.vianetworks.co.uk > > > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 4: Don't 'kill -9' the postmaster > > > > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup. | Newtown Square, Pennsylvania 19073 > ----- Graeme Hinchliffe BSc (Hons) _ Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) - against HTML email X VIA NET.WORKS & vCards / \ http://arc.pasp.de/ Tel +44-1925-484065 Fax +44-1925-484055 ICQ : 3842605 (name : Link) www.vianetworks.co.uk
Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > but dafs isn't nfs. we were trialing netapp for storage but are going > back to local disks as the db ran VERY slowly in comparison. dafs should > accelerate things from what i have read. If it presents a POSIX filesystem API then PG should work OK with it. -Doug
On 30 Oct 2002 08:47:18 -0500 Doug McNaught <doug@mcnaught.org> wrote: > Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > > > but dafs isn't nfs. we were trialing netapp for storage but are > > going back to local disks as the db ran VERY slowly in comparison. > > dafs should accelerate things from what i have read. > > If it presents a POSIX filesystem API then PG should work OK with it. > > -Doug Doesn't appear to. A quick scan of the SDK docs appears as though as if its a direct implementation to access the storage of a device supporting it by applications, such as database servers over a network.. Kinda like raw devices.
Vincent Janelle <random@goblinstudios.com> writes: > On 30 Oct 2002 08:47:18 -0500 > Doug McNaught <doug@mcnaught.org> wrote: > > > Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > > > > > but dafs isn't nfs. we were trialing netapp for storage but are > > > going back to local disks as the db ran VERY slowly in comparison. > > > dafs should accelerate things from what i have read. > > > > If it presents a POSIX filesystem API then PG should work OK with it. > > > > -Doug > > Doesn't appear to. A quick scan of the SDK docs appears as though as if > its a direct implementation to access the storage of a device supporting > it by applications, such as database servers over a network.. Kinda like > raw devices. Well then I can safely say it won't work with PG as distributed. ;) -Doug
On Wed, 30 Oct 2002, Vincent Janelle wrote: > On 30 Oct 2002 08:47:18 -0500 > Doug McNaught <doug@mcnaught.org> wrote: > > > Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > > > > > but dafs isn't nfs. we were trialing netapp for storage but are > > > going back to local disks as the db ran VERY slowly in comparison. > > > dafs should accelerate things from what i have read. > > > > If it presents a POSIX filesystem API then PG should work OK with it. > > > > -Doug > > Doesn't appear to. A quick scan of the SDK docs appears as though as if > its a direct implementation to access the storage of a device supporting > it by applications, such as database servers over a network.. Kinda like > raw devices. Look and see if there's some code out there for your OS (Linux???) to mount a remote network device like this in loop back mode. Then you might be able to let the OS turn it into a file system for the database, which would get you caching on the database server box at the file system level, but block access across the network for speed. Then test it as thouroughly as an Apollo mission. :-)
*shrugs*, I use shared attached storage. Routing this stuff through nscd (which is what most of those user-space apps do) would be silly. You'd still need a filesystem on top of it for postgres, which negates the whole issue. You'd might was well just use a SAN then. Netapp announced that they're adding an option for this, the original poster might want to look into that. On Wed, 30 Oct 2002 15:15:01 -0700 (MST) "scott.marlowe" <scott.marlowe@ihs.com> wrote: > On Wed, 30 Oct 2002, Vincent Janelle wrote: > > > On 30 Oct 2002 08:47:18 -0500 > > Doug McNaught <doug@mcnaught.org> wrote: > > > > > Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > > > > > > > but dafs isn't nfs. we were trialing netapp for storage but are > > > > going back to local disks as the db ran VERY slowly in > > > > comparison. dafs should accelerate things from what i have read. > > > > > > If it presents a POSIX filesystem API then PG should work OK with > > > it. > > > > > > -Doug > > > > Doesn't appear to. A quick scan of the SDK docs appears as though > > as if its a direct implementation to access the storage of a device > > supporting it by applications, such as database servers over a > > network.. Kinda like raw devices. > > Look and see if there's some code out there for your OS (Linux???) to > mount a remote network device like this in loop back mode. > > Then you might be able to let the OS turn it into a file system for > the database, which would get you caching on the database server box > at the file system level, but block access across the network for > speed. > > Then test it as thouroughly as an Apollo mission. :-)
Vincent, did you actually READ my post before writing yours? I said quite clearly that the device would be MOUNTED. I.e. it would have a file system on it. My point being that if you remote mount it as a block device, then the file system on the local machine caches (i.e. fast access) while the remote machine provides (semi-)fast block access across the network. diversity is a good thing. On Wed, 30 Oct 2002, Vincent Janelle wrote: > *shrugs*, I use shared attached storage. Routing this stuff through > nscd (which is what most of those user-space apps do) would be silly. > > You'd still need a filesystem on top of it for postgres, which negates > the whole issue. You'd might was well just use a SAN then. Netapp > announced that they're adding an option for this, the original poster > might want to look into that. > > On Wed, 30 Oct 2002 15:15:01 -0700 (MST) > "scott.marlowe" <scott.marlowe@ihs.com> wrote: > > > On Wed, 30 Oct 2002, Vincent Janelle wrote: > > > > > On 30 Oct 2002 08:47:18 -0500 > > > Doug McNaught <doug@mcnaught.org> wrote: > > > > > > > Graeme Hinchliffe <graeme@vianetworks.co.uk> writes: > > > > > > > > > but dafs isn't nfs. we were trialing netapp for storage but are > > > > > going back to local disks as the db ran VERY slowly in > > > > > comparison. dafs should accelerate things from what i have read. > > > > > > > > If it presents a POSIX filesystem API then PG should work OK with > > > > it. > > > > > > > > -Doug > > > > > > Doesn't appear to. A quick scan of the SDK docs appears as though > > > as if its a direct implementation to access the storage of a device > > > supporting it by applications, such as database servers over a > > > network.. Kinda like raw devices. > > > > Look and see if there's some code out there for your OS (Linux???) to > > mount a remote network device like this in loop back mode. > > > > Then you might be able to let the OS turn it into a file system for > > the database, which would get you caching on the database server box > > at the file system level, but block access across the network for > > speed. > > > > Then test it as thouroughly as an Apollo mission. :-)
I've created this elaborate scheme to create small unique identifiers for
external transactions. It is built on a compact representation of the
current_timestamp(6) + 2 peices of data (varchar(50)) + a sequence number. I'd
like the option of restarting the sequence at some later date, but I don't want
to replicate any IDs.
To shorten the identifier I thought it might be a good idea to run an MD5
digest using the functionality from contrib/pgcrypto on it. ("thought" and
"might" are the key words there...)
I just need another (thousand?) set of eyes to see if a) I've missed anything,
b) There's a better way... :)
-- Sequence: universal_sq
CREATE SEQUENCE universal_sq INCREMENT 1 MINVALUE 1 MAXVALUE
9223372036854775807 CACHE 1;
-- Creates an text/interger representation of the current time
-- with microseconds.
CREATE FUNCTION "dtid"() RETURNS "text" AS '
DECLARE
tstz timestamptz;
dtid text;
BEGIN
tstz := current_timestamp;
dtid := (((tstz::ABSTIME::INT4)/100 * 100000000::INT8) +
date_part(\'microseconds\',tstz))::INT8::TEXT;
RETURN dtid;
END;' LANGUAGE 'plpgsql';
SELECT encode(digest(dtid() || 'value1' || 'value2' ||
nextval('universal_sq')::text, 'md5'),'hex');
__________________________________________________
Do you Yahoo!?
HotJobs - Search new jobs daily now
http://hotjobs.yahoo.com/
For those interested in learning more about what DAFS is and what's been done with it so far, here are the links I've found http://www.dafscollaborative.org http://www.eecs.harvard.edu/~vino/fs-perf/dafs/ http://www.cs.duke.edu/ari/dafs/ It appears there's been work to build in DAFS support for both Linux and BSD, but there's not a lot of data on how far that support has gotten. Basically, it looks like a SAN type of protocol.
and I read the DAFS SDK, which didn't make claims to that. If you mount it as a block device, then you can put a filesystem on it. Problem solved. On Thu, 31 Oct 2002 09:06:04 -0700 (MST) "scott.marlowe" <scott.marlowe@ihs.com> wrote: > Vincent, did you actually READ my post before writing yours? > > I said quite clearly that the device would be MOUNTED. I.e. it would > have a file system on it. > > My point being that if you remote mount it as a block device, then the > > file system on the local machine caches (i.e. fast access) while the > remote machine provides (semi-)fast block access across the network. > > diversity is a good thing. >
Interesting... I've been using DB2 7.2 on Linux on both local disk and with a NetApp and have seen a performance increase going to the NetApp for storage. We have a gigabit network connection between the two, but I'd have thought that local disk would still be faster, but it wasn't. On Wed, 30 Oct 2002, Graeme Hinchliffe wrote: > but dafs isn't nfs. we were trialing netapp for storage but are going > back to local disks as the db ran VERY slowly in comparison. dafs should > accelerate things from what i have read. > > graeme > > On Tue, 29 Oct 2002, Bruce Momjian wrote: > > > > > I know some folks are using NetApp and haven't had any problems. We > > don't officially support it because we discourage NFS because the file > > semantics are not the same as Unix file semantics. > > > > --------------------------------------------------------------------------- > > > > ghinchliffe@vianetworks.co.uk wrote: > > > Hiya > > > We use NetApp filers for network attached storage, these support > > > the DAFS protocol. Is there any support or plan for support of this in > > > PostgreSQL? Would be VERY handy if it was supported. > > > > > > DAFS info can be obtained at: > > > > > > http://www.dafscollaborative.org > > > > > > ----- > > > Graeme Hinchliffe BSc (Hons) _ > > > Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) > > > - against HTML email X > > > VIA NET.WORKS & vCards / \ > > > http://arc.pasp.de/ > > > Tel +44-1925-484065 > > > Fax +44-1925-484055 > > > > > > ICQ : 3842605 (name : Link) > > > > > > www.vianetworks.co.uk > > > > > > > > > ---------------------------(end of broadcast)--------------------------- > > > TIP 4: Don't 'kill -9' the postmaster > > > > > > > -- > > Bruce Momjian | http://candle.pha.pa.us > > pgman@candle.pha.pa.us | (610) 359-1001 > > + If your life is a hard drive, | 13 Roberts Road > > + Christ can be your backup. | Newtown Square, Pennsylvania 19073 > > > > ----- > Graeme Hinchliffe BSc (Hons) _ > Unix Systems Engineer (UK Datacentre) ASCII ribbon campaign ( ) > - against HTML email X > VIA NET.WORKS & vCards / \ > http://arc.pasp.de/ > Tel +44-1925-484065 > Fax +44-1925-484055 > > ICQ : 3842605 (name : Link) > > www.vianetworks.co.uk > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > Dave Thompson dave@guppi.net