Re: RFC: PostgreSQL Storage I/O Transformation Hooks - Mailing list pgsql-hackers

Subject: Re: RFC: PostgreSQL Storage I/O Transformation Hooks

Hi Tomas,

Thank you for this critical feedback. Your concerns go to the heart of the proposal's viability, and I appreciate your directness.


1. Multiple Extensions and Hook Chaining

You're right to question this. To be honest, I have significant doubts about allowing multiple transformation extensions simultaneously.

The Transform ID coordination problem is real: without a registry or protocol between extensions, they cannot cooperate safely. Hook chaining for read/write operations might work (extension A encrypts, extension B compresses), but the Transform ID field creates conflicts.

Perhaps I should be more direct: transformation hook chaining is not realistically possible with the current design. TDE extensions would need exclusive use of these hooks. This is a fundamental limitation I should have stated clearly in the RFC.


2. pd_flags Reservation - I Hope You'll Consider This

I understand your concern about reserving pd_flags bits for extensions. However, I'd like to ask you to consider the reasoning behind this choice.

The 5-bit Transform ID serves a critical purpose: it allows the core to identify the page's transformation state without attempting decryption. This is important for:

- Error reporting: "This page is encrypted with transform ID 5, but no extension is loaded to handle it"
- Migration safety: Distinguishing between untransformed pages (ID=0) and transformed pages during gradual encryption
- Crash recovery: The core can detect transformation state inconsistencies

That said, I recognize pd_flags is precious and limited. Let me propose an alternative approach that might better align with core principles:

Instead of extension-specific Transform IDs, what if we allow extensions to reserve space at pd_upper (similar to how special space works at pd_special)?

The core could manage a small flag (2-3 bits) indicating "N bytes at pd_upper are reserved for transformation metadata". By encoding N as multiples of 2 or 4 bytes, we maximize the flag's efficiency:

- 2 bits encoding 4-byte multiples: 0-12 bytes (sufficient for most cases)
- 3 bits encoding 4-byte multiples: 0-28 bytes (covers all reasonable needs)
- 3 bits encoding 2-byte multiples: 0-14 bytes (finer granularity)

This approach uses minimal pd_flags bits while providing substantial metadata space. It would:

- Keep the flag in core control (not extension-specific)
- Allow extensions to store IV, authentication tags, key version, etc. in a standardized location
- Be self-describing (the flag tells you how much space is reserved)
- Generalize beyond encryption (compression, checksums, etc. could use it)

In our internal implementation, we actually add opaque bytes to PageHeader for encryption metadata. This pd_upper approach could formalize that pattern for extensions.

I believe some form of page-level metadata for transformations is necessary. Would either approach (Transform ID or pd_upper reservation) be acceptable with the right design, or do you see fundamental issues with page-level transformation metadata itself?


3. Maintenance Burden and Test Coverage

I deeply appreciate this concern. Having worked across various DBMS implementations, I've seen solution vendors ship without comprehensive regression testing - but never a database vendor. DBMS maintenance is extraordinarily difficult, and storage errors are catastrophic.

This is precisely why test_tde exists as a reference implementation. But you've identified the real issue: we need much stronger test coverage for the hooks themselves.

The test cases should:
- Detect when core changes break hook contracts
- Verify hook behavior under all I/O paths (sync, async, error cases)
- Validate critical section safety
- Test interaction with checksums, crash recovery, replication

I agree the current test coverage is insufficient for core inclusion. Would expanding the test suite to cover these scenarios address your maintenance concerns, or do you see fundamental fragility beyond what testing can solve?


4. Hooks vs Transform Layer - Pragmatic Timeline

You suggested improving SMGR extensibility rather than adding hooks. I think you're architecturally right about the long-term direction.

However, I want to be pragmatic about timelines:

The hook and pd_flags approach, despite its limitations, can deliver working TDE in the shortest time. Organizations facing regulatory deadlines need something that works now, not in 2-3 years.

That said, your feedback has sparked a better idea: what if we think of this not as "SMGR extension" or "hooks" but as a pluggable Transform Layer that SMGR and WAL subsystems delegate to?

Conceptually:

    Application Layer
           |
    Buffer Manager
           |
    +------------------+
    | Transform Layer  | <-- Encryption, etc.
    +------------------+
           |
      SMGR / WAL
           |
       File I/O

This is architecturally cleaner than scattered hooks, and more focused than full SMGR extensibility. The Transform Layer would:

- Provide a unified interface for data transformation
- Work across backend, frontend tools, and replication
- Handle metadata management in a standardized way
- Support encryption, compression, or other transformations

I think this deserves its own discussion thread rather than conflating it with the current hook proposal. Would you be interested in starting a separate conversation about designing a Transform Layer interface for PostgreSQL?

In the meantime, the hook approach could serve organizations with immediate needs, and extensions could migrate to the Transform Layer once it's stabilized.


5. Frontend Tool Access

Both SMGR and hook approaches face a shared limitation: frontend tools (pg_checksums, pg_basebackup, etc.) that read files directly.

I previously suggested allowing initdb to specify a shared library that both backend and frontend can load for transformation. But as I reconsider this, it feels like it converges toward the Transform Layer idea: a well-defined interface that any PostgreSQL component can use.

This might be the real architectural question: not "hooks vs SMGR" but "how should PostgreSQL provide transformation points that work across backend, frontend, and replication boundaries?"


Summary

Your feedback has clarified three important points:

1. The current hook design has real limitations (multiple extension conflicts, pd_flags concerns)
2. Test coverage needs to be much more comprehensive
3. A cleaner abstraction might be needed long-term

I propose a dual approach:

Short-term: Move forward with the hook proposal for organizations with immediate regulatory needs. I commit to:
- Stating clearly that hook chaining is not supported
- Significantly expanding test coverage
- Treating this as a pragmatic solution with known limitations

Long-term: I'd like to start a separate discussion about a Transform Layer abstraction - a unified interface that could handle data transformation across backend, frontend tools, and replication. This would be architecturally cleaner than scattered hooks, and could eventually supersede this approach.

Would you be willing to review a Transform Layer proposal in a separate thread? I think it addresses the architectural concerns you've raised, while the hook approach serves immediate practical needs.

Best regards,
Henson