Re: Pluggable toaster - Mailing list pgsql-hackers
| From | Nikita Malakhov |
|---|---|
| Subject | Re: Pluggable toaster |
| Date | |
| Msg-id | CAN-LCVMhgN-p9-K87jD0s+kdFXr7gksj96CS9iD7LdrghAHLdQ@mail.gmail.com Whole thread Raw |
| In response to | Re: Pluggable toaster (Aleksander Alekseev <aleksander@timescale.com>) |
| Responses |
Re: Pluggable toaster
(Nikita Malakhov <hukutoc@gmail.com>)
|
| List | pgsql-hackers |
Hi hackers!
--
I want to bump this thread and remind the community about Pluggable TOAST.
Overall Pluggable TOAST was revised to work without altering PG_ATTRIBUTE table
- no atttoaster column, allow to drop unused Toasters and some other improvements.
Sorry for the big delay, but it was a big piece of work to do, and the work is still going on.
Here are the main highlights:
1) No need to modify the PG_ATTRIBUTE table. We've introduced new catalog
table with a set of internal support functions that keeps all table-toaster relations:
postgres@postgres=# \d+ pg_toastrel;
Table "pg_catalog.pg_toastrel"
Column | Type | Collation | Nullable | Default | Storage | Toaster | Compression | Stats target | Description
--------------+----------+-----------+----------+---------+---------+---------+-------------+--------------+-------------
oid | oid | | not null | | plain | | | |
toasteroid | oid | | not null | | plain | | | |
relid | oid | | not null | | plain | | | |
toastentid | oid | | not null | | plain | | | |
attnum | smallint | | not null | | plain | | | |
version | smallint | | not null | | plain | | | |
relname | name | | not null | | plain | | | |
toastentname | name | | not null | | plain | | | |
flag | "char" | | not null | | plain | | | |
toastoptions | "char" | | not null | | plain | | | |
Indexes:
"pg_toastrel_oid_index" PRIMARY KEY, btree (oid)
"pg_toastrel_name_index" UNIQUE CONSTRAINT, btree (toasteroid, relid, version, attnum)
"pg_toastrel_rel_index" btree (relid, attnum)
"pg_toastrel_tsr_index" btree (toasteroid)
Access method: heap
Table "pg_catalog.pg_toastrel"
Column | Type | Collation | Nullable | Default | Storage | Toaster | Compression | Stats target | Description
--------------+----------+-----------+----------+---------+---------+---------+-------------+--------------+-------------
oid | oid | | not null | | plain | | | |
toasteroid | oid | | not null | | plain | | | |
relid | oid | | not null | | plain | | | |
toastentid | oid | | not null | | plain | | | |
attnum | smallint | | not null | | plain | | | |
version | smallint | | not null | | plain | | | |
relname | name | | not null | | plain | | | |
toastentname | name | | not null | | plain | | | |
flag | "char" | | not null | | plain | | | |
toastoptions | "char" | | not null | | plain | | | |
Indexes:
"pg_toastrel_oid_index" PRIMARY KEY, btree (oid)
"pg_toastrel_name_index" UNIQUE CONSTRAINT, btree (toasteroid, relid, version, attnum)
"pg_toastrel_rel_index" btree (relid, attnum)
"pg_toastrel_tsr_index" btree (toasteroid)
Access method: heap
(This is not final definition)
This approach allows us to keep all Toaster assignment history, as well as correctly storing
Toasters assigned to different columns of the relation, and even have separate TOAST
storage entities (these not necessary to be regular TOAST tables) for different columns.
When the table with the TOASTable column is created - a new row is inserted into
PG_TOASTREL with source table OID, Toaster OID, created TOAST entity OID, column
(attribute) index. Special field "version" is used to keep history of Toasters assigned to
the column - it is a counter which increases with each assignment, and the biggest version
is the current Toaster for the column. All assigned Toasters are automatically cached,
and when the value is TOASTed - first lookup is done in cache, and if there is no cached
Toaster it is searched in PG_TOASTREL and inserted in cache.
2) Attribute "reltoastrelid" was replaced with calls of PG_TOASTREL support functions.
This was done to allow each TOASTed column to be assigned with different Toaster
and have its individual TOAST table.
3) DROP TABLE command was modified to remove corresponding records from the
PG_TOASTREL - to allow dropping toasters that are out of use.
4) DROP TOASTER command was introduced. This command allows to drop unused
Toasters - the ones that do not have records in PG_TOASTREL. If the Toaster was
assigned to a column - it could not be dropped, because all data TOASTed with it will
be lost.
The branch is still in development so I it is too early for patch but here's link to the repo:
On Mon, Nov 7, 2022 at 1:35 PM Aleksander Alekseev <aleksander@timescale.com> wrote:
Hi Nikita,
> Pluggable TOAST is provided as an interface to allow developers to plug
> in custom TOAST mechanics. It does not determines would it be universal
> Toaster or one data type, but syntax for universal Toaster is out of scope
> for this patchset.
If I understand correctly, what is going to happen - the same
interface TsrRoutine is going to be used for doing two things:
1) Implementing type-aware TOASTers as a special case for the default
TOAST algorithm when EXTERNAL storage strategy is used.
2) Implementing universal TOASTers from scratch that have nothing to
do with the default TOAST algorithm.
Assuming this is the case, using the same interface for doing two very
different things doesn't strike me as a great design decision. While
working on v24 you may want to rethink this.
Personally I believe that Pluggable TOASTers should support only case
#2. If there is a need of reusing parts of the default TOASTer,
corresponding pieces of code should be declared as non-static and
called from the pluggable TOASTers directly.
Alternatively we could have separate interfaces for case #1 and case
#2 but this IMO becomes rather complicated.
> I'm currently working on a revision of Pluggable TOAST that would make
> dropping Toaster possible if there is no data TOASTed with it, along with
> several other major changes. It will be available in this (I hope so) or the
> following, if I won't make it in time, commitfest.
Looking forward to v24!
This is a major change so I hope there will be more feedback from
other people on the mailing list.
--
Best regards,
Aleksander Alekseev
pgsql-hackers by date: