Re: External data files possible? - Mailing list pgsql-hackers

From Andres Freund
Subject Re: External data files possible?
Date
Msg-id 20220221220836.xjrrneygwdw6xe2n@alap3.anarazel.de
Whole thread Raw
In response to External data files possible?  (Chris Cleveland <ccleve+github@dieselpoint.com>)
List pgsql-hackers
Hi,

On 2022-02-21 15:16:31 -0600, Chris Cleveland wrote:
> It's turning out to be difficult to store the data for my custom index
> access method in the main fork. Breaking up the data into pages with page
> headers means a lot of extra work, a big performance hit, and disk space
> management headaches. It's just not a good fit for my particular file
> format.

I assume you're planning to not go through shared buffers, right?


> It would be much better to store the index in a set of external data files.
> This seems possible so long as I put the files under the database's
> directory and name things properly.
> 
> But here's the one thing I haven't figured out: how to delete the files
> when the index, table, or database gets dropped. The IndexAmRoutine does
> not have an "amdrop" hook that gets called when the index gets dropped.

For some things it'd probably work to just use the normal files, but format
them differently. I.e. go through the smgr.c layer, but not bufmgr.

But unfortunately e.g. basebackup.c will assume they're the normal format and
complain about checksums etc. I don't think there's a way around that right
now.


> Is there a hook I can use to clean these files up? More generally, can I
> get away with using my own data files without causing a problem?

Not currently. A plain hook wouldn't suffice, because it'd not integrate with
transactional DDL and crash recovery.

Greetings,

Andres Freund



pgsql-hackers by date:

Previous
From: Florin Irion
Date:
Subject: Re: Emit a warning if the extension's GUC is set incorrectly
Next
From: "Joel Jacobson"
Date:
Subject: Re: List of all* PostgreSQL EXTENSIONs in the world