On Tue, Jul 08, 2025 at 08:54:33PM +0200, Hannu Krosing wrote:
> I still think we should go with direct toast tid pointers in varlena
> and not some kind of oid.
>
> It will remove the need for any oid management and also will be
> many-many orders of magnitude faster for large tables (just 2x faster
> for in-memory small tables)
There is also the point of backward-compatibility. We cannot just
replace things, and we have to provide a way for users to be able to
rely on the system so as upgrades are painless. So we need to think
about the correct application layer to use to maintain the legacy code
behavior while considering improvements.
> I plan to go over Michael's patch set here and see how much change is
> needed to add the "direct toast"
If you do not have a lot of time looking at the full patch set, I'd
recommend looking at 0003, files toast_external.h and
toast_external.c which include the key idea. Adding a new external
TOAST pointer is then a two-step process:
- Add a new vartag_external.
- Add some callbacks to let the backend understand what it should do
with this new vartag_external.
> My goals are:
> 1. fast lookup from skipping index lookup
> 2. making the toast pointer in main heap as small as possible -
> hopefully just the 6 bytes of tid pointer - so that scans that do not
> need toasted values get more tuples from each page
> 3. adding all (optional) the extra data into toast chunk record as
> there we are free to add whatever is needed
> Currently I plan to introduces something like this for toast chunk record
Points 2. and 3. are things that the refactoring should allow. About
1., I have no idea how much you want to store in the TOAST external
points and how it affects the backend, but you could surely implement
an option that lets the backend know that it should still index
lookups based on what the external TOAST pointer says, if this stuff
has benefits.
> This seems to be flexible enough to allow for both compressin and
> efficient partial updates
I don't really disagree with all that. Now the history of the TOAST
threads point out that we've good at proposing complex things, but
these had a high footprint. What I'm proposing is lighter than that,
I think, tackling my core issue with the infra supporting backward
compatibility and the addition of more modes on top of it.
--
Michael
