F.53. pgpro_iheap — a table access method with an integrated BRIN index #

F.53.1. Description #

The pgpro_iheap extension provides the iheap (indexed heap) table access method that is based on the built-in heap access method and integrates a BRIN index.

The iheap access method is designed to accelerate table scans when the planner selects a sequential scan plan. This scenario typically occurs when no suitable indexes are available for the query. A BRIN index is built into the iheap access method and is not user-controllable. In the rest of the document, the BRIN index integrated within iheap is referred to as the iheap index.

The iheap index is stored in a dedicated index fork of a table. This fork is an additional storage structure, analogous to the visibility map and the free space map. Within this fork, a group of BRIN indexes defined on columns is created for every N pages of the table. The BRIN index group stores the minimum and maximum values and is created for all compatible table columns. Compatible columns are those with fixed or variable length not exceeding 64 bytes. The BRIN index is not utilized for other column types. The number of pages per BRIN index group is configured using the pages_per_index index storage parameter. The default value is 256, and it can be set to any integer between 64 and 512, inclusive. A single iheap index page can store multiple BRIN index groups. The number of groups per page is determined dynamically based on the total size of indexed table columns. Consequently, one iheap page can store index information for tens or even hundreds of thousands of table pages with the standard heap access method.

When a sequential scan is executed, the system evaluates scan conditions against the BRIN index group associated with the iheap index columns prior to reading table data. If the conditions are not satisfied by the index group, the corresponding table block consisting of the number of pages set in pages_per_index is completely skipped during read operations. Skipping such blocks reduces the amount of data read from disk, thereby improving sequential scan performance.

The iheap index is most effective in the following scenarios:

  • As a faster alternative to the standard heap access method for sequential scans when a B-tree index cannot be created, for example, due to insufficient disk space. In such cases, the iheap index can be thousands or even tens of thousands of times smaller than the B-tree index.

  • For queries that use the <, <=, >, and >= comparison operators and target large sample subsets (typically 10–15% of a table) based on the predicate, in situations where a B-tree index cannot be utilized.

  • For indexing monotonically increasing or decreasing column values, as well as values that vary per group of data pages, where the number of pages per group is configured using the pages_per_index index storage parameter. These characteristics make such values suitable for filtering. This scenario most commonly applies to tables whose rows include an insertion timestamp or an incrementing unique ID, for example, logs or tables containing historical data. In these cases, using iheap is preferable to having multiple B-tree indexes for the following reasons:

    • iheap is thousands or even tens of thousands of times smaller than the B-tree index.

    • Unlike the B-tree index, iheap indexes all compatible table columns.

    • Although search performance with iheap is generally slower than with the B-tree index, it is tens of times faster than with the standard heap access method.

F.53.2. Considerations and Limitations #

  • When using the iheap access method, only columns with fixed or variable length are indexed, provided that the length does not exceed 64 bytes. For columns of the varlena data type, this size includes the column header.

  • The column data type must be supported by the B-tree index, as the iheap access method uses the comparison operators of the B-tree index, given its status as the most widely used index access method.

  • A BRIN index group defined on all table columns must fit into a single iheap index page, which approximately corresponds to a single table with 1,000 columns of the integer data type. If there is not enough space to store the column data, an error message is returned. The message includes the number of bytes that were attempted to be allocated for iheap index columns.

  • The pages_per_index index storage parameter can only be changed by following these steps:

    1. Change the access method from iheap to heap using the iheap_set_am function.

    2. Change the pages_per_index value.

    3. Change the access method back to iheap by calling iheap_set_am again.

    The iheap index is then rebuilt with the new index storage parameter value.

  • When DELETE or UPDATE operations are performed, data on table pages changes, and the corresponding iheap index groups are marked as obsolete. Although data from these groups remains accessible, the index groups must be updated by executing VACUUM or by relying on autovacuum.

    For tables using the iheap access method, the VACUUM command, when executed with the disabled DISABLE_PAGE_SKIPPING parameter, processes only those pages that belong to iheap index groups marked as obsolete. Autovacuum uses the same approach.

  • When the VACUUM command is executed with the DISABLE_PAGE_SKIPPING parameter enabled, it iterates through all data pages and builds the iheap index for those pages.

  • For the boolean data type, only the column = true conditions are supported.

F.53.3. Installation #

The pgpro_iheap extension is included in Postgres Pro Enterprise. To enable pgpro_iheap, create the extension using the following query:

CREATE EXTENSION pgpro_iheap;

F.53.4. Functions #

The pgpro_iheap extension provides the following functions:

iheap_set_am(rel_name text, am_name text) returns bool #

Changes the table access method, for example, from iheap to heap or vice versa. When changing from heap to iheap, the iheap index is built. When the access method is changed from iheap to heap, the iheap index is dropped. Thus, the function works similarly to the CREATE INDEX and DROP INDEX commands.

Note that the iheap index, i.e., the table index fork, is created or dropped only after the transaction is committed, not immediately upon calling the iheap_set_am function. This behavior eliminates the need to roll back changes in the event of an error. Consequently, when changing the table access method from heap to iheap, the table can be accessed only in the subsequent transaction.

After calling this function, the PREPARE TRANSACTION command fails with an error.

iheap_page_info(rel_name text, blk_num int4) returns record #

Returns general information about the blk_num page of the iheap index for the table rel_name. The output contains the following columns:

  • col_group_size: The size of a BRIN index group defined on a column.

  • col_groups_num: The number of BRIN index groups on an iheap page.

iheap_page_groups_info(rel_name text, blk_num int4) returns setof record #

Returns BRIN index groups defined on columns of the blk_num page of the iheap index for the table rel_name. The output contains the following columns:

  • group_num: The group number starting from 0.

  • flags: The flags set for the group:

    • set: The group contains values of BRIN indexes defined on columns, meaning that the group is not empty.

    • obsolete: The group data became outdated and will be updated when the vacuum is performed.

iheap_page_items_info(rel_name text, blk_num int4, group_num int4) returns setof record #

Returns the contents of the group_num BRIN index group of the blk_num page of the iheap index for the table rel_name. The output contains the following columns:

  • attno: The number of the indexed column starting from 1.

  • flags: The flags set for the column:

    • set: The column has BRIN index values.

    • has nulls: The column has one or more NULL values.

  • minval: The minimum BRIN index value.

  • maxval: The maximum BRIN index value.

F.53.5. Usage #

In the following example, the extension is created, a table with the iheap access method is created, and various operations are performed on that table.

-- Create the pgpro_iheap extension
CREATE EXTENSION pgpro_iheap;

-- Create a table with the iheap access method
CREATE TABLE test (i int, t text, b bigint) USING iheap;

-- Add data (NULL is added to the third column in the first page)
INSERT INTO test (i, t, b) VALUES (1, '1', NULL);
INSERT INTO test (SELECT i, i::text, i % 100 FROM generate_series(2, 100000) AS i);

-- Retrieve headers of the iheap index pages
SELECT i AS blkno, (info).col_group_size, (info).col_groups_num FROM
  (SELECT i, iheap_page_info('test', i) AS info FROM
    (SELECT i FROM generate_series(0, ((pg_relation_size('test', 'idx') / 8192) - 1)::int) AS i));

 blkno | col_group_size | col_groups_num
-------+----------------+----------------
     0 |             32 |            255
(1 row)

-- Retrieve information about the iheap index groups
SELECT i AS blkno, (info).group_num, (info).flags FROM
  (SELECT i, iheap_page_groups_info('test', i) AS info FROM
    (SELECT i FROM generate_series(0, ((pg_relation_size('test', 'idx') / 8192) - 1)::int) AS i))
  WHERE array_position((info).flags, 'set') > 0;

 blkno | group_num | flags
-------+-----------+-------
     0 |         0 | {set}
     0 |         1 | {set}
     0 |         2 | {set}
(3 rows)

-- Retrieve information about columns of the iheap index groups
SELECT i AS blkno, group_num, (info).attno, (info).flags FROM
  (SELECT i, group_num, iheap_page_items_info('test', i, group_num) AS info FROM
    (SELECT i, (info).group_num AS group_num FROM
      (SELECT i, iheap_page_groups_info('test', i) AS info FROM
        (SELECT i FROM generate_series(0, ((pg_relation_size('test', 'idx') / 8192) - 1)::int) AS i))
      WHERE array_position((info).flags, 'set') > 0));

 blkno | group_num | attno |       flags
-------+-----------+-------+-------------------
     0 |         0 |     1 | {set}
     0 |         0 |     2 | {}
     0 |         0 |     3 | {set,"has nulls"}
     0 |         1 |     1 | {set}
     0 |         1 |     2 | {}
     0 |         1 |     3 | {set}
     0 |         2 |     1 | {set}
     0 |         2 |     2 | {}
     0 |         2 |     3 | {set}
(9 rows)

-- Search for a value using the iheap index
SELECT * FROM test WHERE i = 99999;

   i   |   t   | b
-------+-------+----
 99999 | 99999 | 99
(1 row)