Re: PostgreSQL as a filesystem - Mailing list pgsql-general

> On Mon, 2005-04-18 at 17:18 -0400, Tom Lane wrote:
> > "Christopher Nelson" <paradox@BBHC.ORG> writes:
> > > I'm developing a hobby OS and I'm looking into file systems.  I've
> > > thought about writing my own, and that appeals, but I'm also very
> > > interested in the database-as-a-filesystem paradigm.  It would be
nice
> > > to not have to write all of the stuff that goes into the DBMS
(e.g.
> > > parsers, query schedulers, etc) myself.
> >
> > > So I was wondering what sort of filesystem requirements Postgre
has.
> >
> > There are DB's you could use for this, but Postgres (not "Postgre",
> > please, there is no such animal) isn't one of them :-(.  We really
> > assume we are sitting on top of a full-spec file system --- we want
> > space management for variable-size files, robust storage of
directory
> > information, etc.
>
> I've been thinking of it, too. I think no filesystem out there is
really
> optimized for a steady write load with many fsyncs, that is, is really
> transaction-oriented on the data side (journalled ones may implement
> real transactions for meta-data, but only for it). Out of curiosity,
> do you have any feedback from filesystem people, are they interested
in
> optimizing for the kind of workload (expecially on write) a database
> generates? I ask for it seems to me it's a corner case to them, or
even
> a degenerated one. I'm not aware of _any_ comparative benchmarch among
> different filesystems that is based on write+fsync load, for one.

I don't know of any filesystem people who have a desire to explicitly
support that sort of traffic.  I have looked at the internals of systems
like BFS, and those journaled systems support transactions for all
data... not just metadata.  For example, on BFS there is an area where
all data is journaled, then once it's been verified that the data
journaling is done, the log is rolled forward.  XFS has an interesting
alternative.  They do only journal metadata, but no filedata is
overwritten until the transaction succeeds.  So what they do is write
the transaction metadata, allocate new storage for the block, write the
block, copy the extents map with the new block, commit the new extents
map, and then commit the metadata.  So during all parts of the process,
up until the final commit of the metadata, two copies of everything
exist for that context.

> Using a DB as filesystem at OS level is a different matter, of course.

Which is what I'm trying to accomplish.

> Christopher, you may have a look at FUSE.
> http://fuse.sourceforge.net/

Thanks for the link.  It's not exactly what I'm looking for, since I'm
using the spoon microkernel and the file system is going to be a user
space agent in any case.  But the information is interesting.

> It may help in both developing a new filesystem and in understanding
> how it works under Linux (with a nice separation of userspace and
> kernelspace). I think you could even write one based on PostgreSQL,
> but it won't help much, since PostgreSQL needs a filesystem to work.
> But if your OS has TCP/IP, it could be interesting anyway.
>
> Note that I'm not aware of any other way to access PostgreSQL than
> sockets, so you need those at least. There's no standalone library
> you can link to in order to access database files, AFAIK.

Hmm. So it would be a LOT of work to use it.  Obviously I wouldn't be
using sockets, but I would be using an IPC primitive similar to sockets.
It would be relatively simple to create a basic filesystem abstraction
that kept track of large blocks of data, and nothing else.  Then mount
the database layer on top of that.

I suppose it would make more sense to have both raw data streams and
associated relational object data.  Streams for data performance, and
the relational data for information about the stream.

-={C}=-