Thread: BLOBs and a virtual file system

BLOBs and a virtual file system

From
Peter Martini
Date:
Lately I've been planning work on a patch to postgres and linux on my
system to allow access to BLOBs as a virtual filesystem, so I can see
any file I put in there even through network shares while avoiding
duplication / broken link issues.  Does this sound like something worth
doing / is there a better way to safely reference files from both inside
and outside postgres?


Re: BLOBs and a virtual file system

From
Tom Lane
Date:
Peter Martini <PeterM@processflo.com> writes:
> Lately I've been planning work on a patch to postgres and linux on my
> system to allow access to BLOBs as a virtual filesystem, so I can see
> any file I put in there even through network shares while avoiding
> duplication / broken link issues.  Does this sound like something worth
> doing / is there a better way to safely reference files from both inside
> and outside postgres?

How would you do that without breaking transactional integrity for
blobs?  There'd be no way to deal with multiple row versions in such
a representation.
        regards, tom lane


Re: BLOBs and a virtual file system

From
"Dave Bauer"
Date:
Peter Martini writes: 

> Lately I've been planning work on a patch to postgres and linux on my
> system to allow access to BLOBs as a virtual filesystem, so I can see
> any file I put in there even through network shares while avoiding
> duplication / broken link issues.  Does this sound like something worth
> doing / is there a better way to safely reference files from both inside
> and outside postgres? 
> 

I have been looking into using the Tcl VFS features for this kind of thing. 
I work with AOLserver and PostgreSQL so I have Tcl and PostgreSQL available 
to me. 

http://nnsa.dl.ac.uk/MIDAS/manual/ActiveTcl8.4.4.0-html/tclvfs/doc/vfs.html 

I haven't thought about accessing the VFS from inside PostgreSQL though. 

Dave Bauer
dave@thedesignexperience.org 


> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match



Re: BLOBs and a virtual file system

From
Peter Martini
Date:
I'm not sure what you mean by breaking blobs transactional integrity. 
Do you mean by allowing filesystem type access, the blobs won't be
properly locked and updated during a transaction?  If so, that's exactly
what I'm trying to achieve - a compromise between forcing the files to
be stored solely in the database (maintaining integrity but sacrificing
functionality) or referencing their names to gain functionality, but at
a greater cost.
More to the point, wouldn't file locking mechanisms, and the possibility
of limiting file visibility through the virtual filesystem, allow
transactional integrity to be maintained?  If not, could you explain
where the problem is so I can look further into it?

Thanks,
Peter

On Wed, 2004-06-23 at 11:48, Tom Lane wrote:
> Peter Martini <PeterM@processflo.com> writes:
> > Lately I've been planning work on a patch to postgres and linux on my
> > system to allow access to BLOBs as a virtual filesystem, so I can see
> > any file I put in there even through network shares while avoiding
> > duplication / broken link issues.  Does this sound like something worth
> > doing / is there a better way to safely reference files from both inside
> > and outside postgres?
> 
> How would you do that without breaking transactional integrity for
> blobs?  There'd be no way to deal with multiple row versions in such
> a representation.
> 
>             regards, tom lane



Re: BLOBs and a virtual file system

From
Frank Wiles
Date:
On Wed, 23 Jun 2004 15:07:42 -0400
Peter Martini <peterm@processflo.com> wrote:

> I'm not sure what you mean by breaking blobs transactional integrity. 
> Do you mean by allowing filesystem type access, the blobs won't be
> properly locked and updated during a transaction?  If so, that's
> exactly what I'm trying to achieve - a compromise between forcing the
> files to be stored solely in the database (maintaining integrity but
> sacrificing functionality) or referencing their names to gain
> functionality, but at a greater cost.
> More to the point, wouldn't file locking mechanisms, and the
> possibility of limiting file visibility through the virtual
> filesystem, allow transactional integrity to be maintained?  If not,
> could you explain where the problem is so I can look further into it?
 I believe the problem is that the blob could be in several different states inside of several different transactions.
Howdo you  determine which you show in the filesystem? 
 
 Even if the file system is read only you still have this problem of "which" of the many possible BLOBs to reveal.  
---------------------------------  Frank Wiles <frank@wiles.org>
http://frank.wiles.org---------------------------------