On 28/12/2025 9:49 AM, Henson Choi wrote:

RFC: PostgreSQL Storage I/O Transformation Hooks

Infrastructure for a Technical Protocol Between RDBMS Core and Data Security Experts

Author: Henson Choi assam258@gmail.com

Date: 2025-12-28

PostgreSQL Version: master (Development)

---

1. Summary & Motivation

This RFC proposes the introduction of minimal hooks into the PostgreSQL storage layer and the addition of a Transformation ID field to the PageHeader.

A Diplomatic Protocol Between Expert Groups

The core motivation of this proposal is “Separation of Concerns and Mutual Respect.”

Historically, discussions around Transparent Data Encryption (TDE) have often felt like putting security experts on trial in a foreign court—specifically, the “Court of RDBMS.” It is time to treat them not as defendants to be judged by database-specific rules, but as an equal neighboring community with their own specialized sovereignty.

The issue has never been a failure of technology, but rather a misplacement of the focal point. While previous discussions were mired in the technicalities of “how to hardcode encryption into the core,” this proposal shifts the debate toward an architectural solution: “what interface the core should provide to external experts.”

• RDBMS Experts provide a trusted pipeline responsible for data I/O paths and consistency.
• Security Experts take responsibility for the specialized domain of encryption algorithms and key management.

This hook system functions as a Technical Protocol—a high-level agreement that allows these two expert groups to exchange data securely without encroaching on each other’s territory.

---

2. Design Principles

1. Delegation of Authority: The core remains independent of specific encryption standards, providing a “free territory” where security experts can respond to an ever-changing security landscape.
2. Diplomatic Convention: The Transformation ID acts as a communication protocol between the engine and the extension. The engine uses this ID to identify the state of the data and hands over control to the appropriate expert (the extension).
3. Minimal Interference: Overhead is kept near zero when hooks are not in use, ensuring the native performance of the PostgreSQL engine.

---

3. Proposal Specifications

3.1 The Interface (Hook Points)

We allow intervention by security experts through five contact points along the I/O path:

• Read/Write Hooks: mdread_post, mdwrite_pre, mdextend_pre (Transformation of the data area)
• WAL Hooks: xlog_insert_pre, xlog_decode_pre (Transformation of transaction logs)

3.2 The Protocol Identifier (PageHeader Transformation ID)

We allocate 5 bits of pd_flags to define the “Security State” of a page. This serves as a Status Message sent by the security expert to the engine, utilized for key versioning and as a migration marker.

---

4. Reference Implementation: contrib/test_tde

A Standard Code of Conduct for Security Experts

This reference implementation exists not as a commercial product, but to define the Standards of the Diplomatic Protocol that encryption/decryption experts must follow when entering the PostgreSQL domain.

1. Deterministic IV Derivation: Demonstrates how to achieve cryptographic safety by trusting unique values provided by the engine (e.g., LSN).
2. Critical Section Safety: Defines memory management regulations that security logic must follow within “Critical Sections” to maintain system stability.
3. Hook Chaining: Demonstrates a cooperative structure that allows peaceful coexistence with other expert tools (e.g., compression, auditing).

---

5. Scope

• In-Scope: Backend hook infrastructure, Transformation ID field, and reference code demonstrating diplomatic protocol compliance.
• Out-of-Scope: Specific Key Management Systems (KMS), selection of specific cryptographic algorithms, and integration with external tools.

This proposal represents a strategic diplomatic choice: rather than the PostgreSQL core assuming all security responsibilities, it grants security experts a sovereign territory through extensions where they can perform at their best.

I wonder if instead of support a lot of extra hooks it will be better to provide extensible SMGR API:
https://www.postgresql.org/message-id/flat/CAPP%3DHha_wV1MV9yR70QZ5pk5dtNP%2BbOyBiFxPmrMKqnQeKMAwQ%40mail.gmail.com#ab0da3412525c7501ea17f3d4c602bbf
It seems to be much more straightforward, convenient and flexible mechanism than adding hooks, which can be used for many other purposes except transparent encryption.