Re: RFC: PostgreSQL Storage I/O Transformation Hooks - Mailing list pgsql-hackers

Subject: Re: RFC: PostgreSQL Storage I/O Transformation Hooks

Hi Zsolt,

Thank you for your detailed questions. I'll address each point:

1. Bundling WAL and Buffer Manager

WAL and heap pages are simply different representations of the same
underlying data. Protecting only one side would be cryptographically
incomplete; an attacker could bypass encryption by reading the
unprotected side. Therefore, they must be treated as a single atomic
unit of protection.

2. Scope: Temporary Files, System Tables, and Frontend Tools

I intentionally kept the scope focused. Past TDE proposals often stalled
because they tried to solve everything at once, becoming too large to
review. I prefer a "divide-and-conquer" approach:

- Temporary files: Out of scope for this initial infrastructure proposal.
- System tables: While they cannot be encrypted during bootstrap (since
  extensions aren't loaded), they can be transformed page-by-page during
  normal operation.
- Frontend tools (pg_waldump, etc.): I am aware of this and have modified
  versions. Currently, there is no standard mechanism for frontend hooks,
  making this a broader challenge. For production, extensions could ship
  their own modified frontend tools temporarily. Long-term, we may need
  initdb-time configurations to unify backend/frontend hook behavior
  that are fixed for the lifetime of the cluster.

3. Why Hooks Instead of SMGR

Please see my response to Konstantin in this thread regarding maintenance
debt and the "Separation of Concerns" between storage management and data
transformation.

4. Page Header Flags vs. Fork Files

My primary concern with using fork files for encryption metadata is crash
recovery. If a fork file and the actual data page become inconsistent
(e.g., during a crash), recovery becomes problematic because fork files
are not typically protected by WAL.

Storing the Transform ID in the header flags ensures that the metadata
travels with the page. This is essential for incremental key rotation,
where pages are gradually re-encrypted with newer keys over time. The
oldest key's pages are force-rotated, allowing continuous key rotation
without service interruption. I plan to propose a separate RFC for this
"gradual rotation" mechanism.

5. Benchmarks and Critical Section Overhead

Transformation happens inside the critical section but before acquiring
the WAL lock. On consumer-grade SSDs, the encryption latency is largely
masked by I/O wait times with negligible performance impact. On
high-performance storage (production SSDs, Apple Silicon, etc.), the
reduced I/O wait exposes the encryption overhead, which is visible but
modest. Detailed benchmarks require company approval - I will follow up
later.

Best regards,
Henson Choi