65.3. Extensibility #
SP-GiST offers an interface with a high level of abstraction, requiring the access method developer to implement only methods specific to a given data type. The SP-GiST core is responsible for efficient disk mapping and searching the tree structure. It also takes care of concurrency and logging considerations.
Leaf tuples of an SP-GiST tree usually contain values of the same data type as the indexed column, although it is also possible for them to contain lossy representations of the indexed column. Leaf tuples stored at the root level will directly represent the original indexed data value, but leaf tuples at lower levels might contain only a partial value, such as a suffix. In that case the operator class support functions must be able to reconstruct the original value using information accumulated from the inner tuples that are passed through to reach the leaf level.
When an SP-GiST index is created with INCLUDE
columns, the values of those columns are also stored in leaf tuples. The INCLUDE
columns are of no concern to the SP-GiST operator class, so they are not discussed further here.
Inner tuples are more complex, since they are branching points in the search tree. Each inner tuple contains a set of one or more nodes, which represent groups of similar leaf values. A node contains a downlink that leads either to another, lower-level inner tuple, or to a short list of leaf tuples that all lie on the same index page. Each node normally has a label that describes it; for example, in a radix tree the node label could be the next character of the string value. (Alternatively, an operator class can omit the node labels, if it works with a fixed set of nodes for all inner tuples; see Section 65.4.2.) Optionally, an inner tuple can have a prefix value that describes all its members. In a radix tree this could be the common prefix of the represented strings. The prefix value is not necessarily really a prefix, but can be any data needed by the operator class; for example, in a quad-tree it can store the central point that the four quadrants are measured with respect to. A quad-tree inner tuple would then also contain four nodes corresponding to the quadrants around this central point.
Some tree algorithms require knowledge of level (or depth) of the current tuple, so the SP-GiST core provides the possibility for operator classes to manage level counting while descending the tree. There is also support for incrementally reconstructing the represented value when that is needed, and for passing down additional data (called traverse values) during a tree descent.
Note
The SP-GiST core code takes care of null entries. Although SP-GiST indexes do store entries for nulls in indexed columns, this is hidden from the index operator class code: no null index entries or search conditions will ever be passed to the operator class methods. (It is assumed that SP-GiST operators are strict and so cannot succeed for null values.) Null values are therefore not discussed further here.
There are five user-defined methods that an index operator class for SP-GiST must provide, and two are optional. All five mandatory methods follow the convention of accepting two internal
arguments, the first of which is a pointer to a C struct containing input values for the support method, while the second argument is a pointer to a C struct where output values must be placed. Four of the mandatory methods just return void
, since all their results appear in the output struct; but leaf_consistent
returns a boolean
result. The methods must not modify any fields of their input structs. In all cases, the output struct is initialized to zeroes before calling the user-defined method. The optional sixth method compress
accepts a datum
to be indexed as the only argument and returns a value suitable for physical storage in a leaf tuple. The optional seventh method options
accepts an internal
pointer to a C struct, where opclass-specific parameters should be placed, and returns void
.
The five mandatory user-defined methods are:
config
Returns static information about the index implementation, including the data type OIDs of the prefix and node label data types.
The SQL declaration of the function must look like this:
CREATE FUNCTION my_config(internal, internal) RETURNS void ...
The first argument is a pointer to a
spgConfigIn
C struct, containing input data for the function. The second argument is a pointer to aspgConfigOut
C struct, which the function must fill with result data.typedef struct spgConfigIn { Oid attType; /* Data type to be indexed */ } spgConfigIn; typedef struct spgConfigOut { Oid prefixType; /* Data type of inner-tuple prefixes */ Oid labelType; /* Data type of inner-tuple node labels */ Oid leafType; /* Data type of leaf-tuple values */ bool canReturnData; /* Opclass can reconstruct original data */ bool longValuesOK; /* Opclass can cope with values > 1 page */ } spgConfigOut;
attType
is passed in order to support polymorphic index operator classes; for ordinary fixed-data-type operator classes, it will always have the same value and so can be ignored.For operator classes that do not use prefixes,
prefixType
can be set toVOIDOID
. Likewise, for operator classes that do not use node labels,labelType
can be set toVOIDOID
.canReturnData
should be set true if the operator class is capable of reconstructing the originally-supplied index value.longValuesOK
should be set true only when theattType
is of variable length and the operator class is capable of segmenting long values by repeated suffixing (see Section 65.4.1).leafType
should match the index storage type defined by the operator class'sopckeytype
catalog entry. (Note thatopckeytype
can be zero, implying the storage type is the same as the operator class's input type, which is the most common situation.) For reasons of backward compatibility, theconfig
method can setleafType
to some other value, and that value will be used; but this is deprecated since the index contents are then incorrectly identified in the catalogs. Also, it's permissible to leaveleafType
uninitialized (zero); that is interpreted as meaning the index storage type derived fromopckeytype
.When
attType
andleafType
are different, the optional methodcompress
must be provided. Methodcompress
is responsible for transformation of datums to be indexed fromattType
toleafType
.choose
Chooses a method for inserting a new value into an inner tuple.
The SQL declaration of the function must look like this:
CREATE FUNCTION my_choose(internal, internal) RETURNS void ...
The first argument is a pointer to a
spgChooseIn
C struct, containing input data for the function. The second argument is a pointer to aspgChooseOut
C struct, which the function must fill with result data.typedef struct spgChooseIn { Datum datum; /* original datum to be indexed */ Datum leafDatum; /* current datum to be stored at leaf */ int level; /* current level (counting from zero) */ /* Data from current inner tuple */ bool allTheSame; /* tuple is marked all-the-same? */ bool hasPrefix; /* tuple has a prefix? */ Datum prefixDatum; /* if so, the prefix value */ int nNodes; /* number of nodes in the inner tuple */ Datum *nodeLabels; /* node label values (NULL if none) */ } spgChooseIn; typedef enum spgChooseResultType { spgMatchNode = 1, /* descend into existing node */ spgAddNode, /* add a node to the inner tuple */ spgSplitTuple /* split inner tuple (change its prefix) */ } spgChooseResultType; typedef struct spgChooseOut { spgChooseResultType resultType; /* action code, see above */ union { struct /* results for spgMatchNode */ { int nodeN; /* descend to this node (index from 0) */ int levelAdd; /* increment level by this much */ Datum restDatum; /* new leaf datum */ } matchNode; struct /* results for spgAddNode */ { Datum nodeLabel; /* new node's label */ int nodeN; /* where to insert it (index from 0) */ } addNode; struct /* results for spgSplitTuple */ { /* Info to form new upper-level inner tuple with one child tuple */ bool prefixHasPrefix; /* tuple should have a prefix? */ Datum prefixPrefixDatum; /* if so, its value */ int prefixNNodes; /* number of nodes */ Datum *prefixNodeLabels; /* their labels (or NULL for * no labels) */ int childNodeN; /* which node gets child tuple */ /* Info to form new lower-level inner tuple with all old nodes */ bool postfixHasPrefix; /* tuple should have a prefix? */ Datum postfixPrefixDatum; /* if so, its value */ } splitTuple; } result; } spgChooseOut;
datum
is the original datum ofspgConfigIn
.attType
type that was to be inserted into the index.leafDatum
is a value ofspgConfigOut
.leafType
type, which is initially a result of methodcompress
applied todatum
when methodcompress
is provided, or the same value asdatum
otherwise.leafDatum
can change at lower levels of the tree if thechoose
orpicksplit
methods change it. When the insertion search reaches a leaf page, the current value ofleafDatum
is what will be stored in the newly created leaf tuple.level
is the current inner tuple's level, starting at zero for the root level.allTheSame
is true if the current inner tuple is marked as containing multiple equivalent nodes (see Section 65.4.3).hasPrefix
is true if the current inner tuple contains a prefix; if so,prefixDatum
is its value.nNodes
is the number of child nodes contained in the inner tuple, andnodeLabels
is an array of their label values, or NULL if there are no labels.The
choose
function can determine either that the new value matches one of the existing child nodes, or that a new child node must be added, or that the new value is inconsistent with the tuple prefix and so the inner tuple must be split to create a less restrictive prefix.If the new value matches one of the existing child nodes, set
resultType
tospgMatchNode
. SetnodeN
to the index (from zero) of that node in the node array. SetlevelAdd
to the increment inlevel
caused by descending through that node, or leave it as zero if the operator class does not use levels. SetrestDatum
to equalleafDatum
if the operator class does not modify datums from one level to the next, or otherwise set it to the modified value to be used asleafDatum
at the next level.If a new child node must be added, set
resultType
tospgAddNode
. SetnodeLabel
to the label to be used for the new node, and setnodeN
to the index (from zero) at which to insert the node in the node array. After the node has been added, thechoose
function will be called again with the modified inner tuple; that call should result in anspgMatchNode
result.If the new value is inconsistent with the tuple prefix, set
resultType
tospgSplitTuple
. This action moves all the existing nodes into a new lower-level inner tuple, and replaces the existing inner tuple with a tuple having a single downlink pointing to the new lower-level inner tuple. SetprefixHasPrefix
to indicate whether the new upper tuple should have a prefix, and if so setprefixPrefixDatum
to the prefix value. This new prefix value must be sufficiently less restrictive than the original to accept the new value to be indexed. SetprefixNNodes
to the number of nodes needed in the new tuple, and setprefixNodeLabels
to a palloc'd array holding their labels, or to NULL if node labels are not required. Note that the total size of the new upper tuple must be no more than the total size of the tuple it is replacing; this constrains the lengths of the new prefix and new labels. SetchildNodeN
to the index (from zero) of the node that will downlink to the new lower-level inner tuple. SetpostfixHasPrefix
to indicate whether the new lower-level inner tuple should have a prefix, and if so setpostfixPrefixDatum
to the prefix value. The combination of these two prefixes and the downlink node's label (if any) must have the same meaning as the original prefix, because there is no opportunity to alter the node labels that are moved to the new lower-level tuple, nor to change any child index entries. After the node has been split, thechoose
function will be called again with the replacement inner tuple. That call may return anspgAddNode
result, if no suitable node was created by thespgSplitTuple
action. Eventuallychoose
must returnspgMatchNode
to allow the insertion to descend to the next level.picksplit
Decides how to create a new inner tuple over a set of leaf tuples.
The SQL declaration of the function must look like this:
CREATE FUNCTION my_picksplit(internal, internal) RETURNS void ...
The first argument is a pointer to a
spgPickSplitIn
C struct, containing input data for the function. The second argument is a pointer to aspgPickSplitOut
C struct, which the function must fill with result data.typedef struct spgPickSplitIn { int nTuples; /* number of leaf tuples */ Datum *datums; /* their datums (array of length nTuples) */ int level; /* current level (counting from zero) */ } spgPickSplitIn; typedef struct spgPickSplitOut { bool hasPrefix; /* new inner tuple should have a prefix? */ Datum prefixDatum; /* if so, its value */ int nNodes; /* number of nodes for new inner tuple */ Datum *nodeLabels; /* their labels (or NULL for no labels) */ int *mapTuplesToNodes; /* node index for each leaf tuple */ Datum *leafTupleDatums; /* datum to store in each new leaf tuple */ } spgPickSplitOut;
nTuples
is the number of leaf tuples provided.datums
is an array of their datum values ofspgConfigOut
.leafType
type.level
is the current level that all the leaf tuples share, which will become the level of the new inner tuple.Set
hasPrefix
to indicate whether the new inner tuple should have a prefix, and if so setprefixDatum
to the prefix value. SetnNodes
to indicate the number of nodes that the new inner tuple will contain, and setnodeLabels
to an array of their label values, or to NULL if node labels are not required. SetmapTuplesToNodes
to an array that gives the index (from zero) of the node that each leaf tuple should be assigned to. SetleafTupleDatums
to an array of the values to be stored in the new leaf tuples (these will be the same as the inputdatums
if the operator class does not modify datums from one level to the next). Note that thepicksplit
function is responsible for palloc'ing thenodeLabels
,mapTuplesToNodes
andleafTupleDatums
arrays.If more than one leaf tuple is supplied, it is expected that the
picksplit
function will classify them into more than one node; otherwise it is not possible to split the leaf tuples across multiple pages, which is the ultimate purpose of this operation. Therefore, if thepicksplit
function ends up placing all the leaf tuples in the same node, the core SP-GiST code will override that decision and generate an inner tuple in which the leaf tuples are assigned at random to several identically-labeled nodes. Such a tuple is markedallTheSame
to signify that this has happened. Thechoose
andinner_consistent
functions must take suitable care with such inner tuples. See Section 65.4.3 for more information.picksplit
can be applied to a single leaf tuple only in the case that theconfig
function setlongValuesOK
to true and a larger-than-a-page input value has been supplied. In this case the point of the operation is to strip off a prefix and produce a new, shorter leaf datum value. The call will be repeated until a leaf datum short enough to fit on a page has been produced. See Section 65.4.1 for more information.inner_consistent
Returns set of nodes (branches) to follow during tree search.
The SQL declaration of the function must look like this:
CREATE FUNCTION my_inner_consistent(internal, internal) RETURNS void ...
The first argument is a pointer to a
spgInnerConsistentIn
C struct, containing input data for the function. The second argument is a pointer to aspgInnerConsistentOut
C struct, which the function must fill with result data.typedef struct spgInnerConsistentIn { ScanKey scankeys; /* array of operators and comparison values */ ScanKey orderbys; /* array of ordering operators and comparison * values */ int nkeys; /* length of scankeys array */ int norderbys; /* length of orderbys array */ Datum reconstructedValue; /* value reconstructed at parent */ void *traversalValue; /* opclass-specific traverse value */ MemoryContext traversalMemoryContext; /* put new traverse values here */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */ /* Data from current inner tuple */ bool allTheSame; /* tuple is marked all-the-same? */ bool hasPrefix; /* tuple has a prefix? */ Datum prefixDatum; /* if so, the prefix value */ int nNodes; /* number of nodes in the inner tuple */ Datum *nodeLabels; /* node label values (NULL if none) */ } spgInnerConsistentIn; typedef struct spgInnerConsistentOut { int nNodes; /* number of child nodes to be visited */ int *nodeNumbers; /* their indexes in the node array */ int *levelAdds; /* increment level by this much for each */ Datum *reconstructedValues; /* associated reconstructed values */ void **traversalValues; /* opclass-specific traverse values */ double **distances; /* associated distances */ } spgInnerConsistentOut;
The array
scankeys
, of lengthnkeys
, describes the index search condition(s). These conditions are combined with AND — only index entries that satisfy all of them are interesting. (Note thatnkeys
= 0 implies that all index entries satisfy the query.) Usually the consistent function only cares about thesk_strategy
andsk_argument
fields of each array entry, which respectively give the indexable operator and comparison value. In particular it is not necessary to checksk_flags
to see if the comparison value is NULL, because the SP-GiST core code will filter out such conditions. The arrayorderbys
, of lengthnorderbys
, describes ordering operators (if any) in the same manner.reconstructedValue
is the value reconstructed for the parent tuple; it is(Datum) 0
at the root level or if theinner_consistent
function did not provide a value at the parent level.traversalValue
is a pointer to any traverse data passed down from the previous call ofinner_consistent
on the parent index tuple, or NULL at the root level.traversalMemoryContext
is the memory context in which to store output traverse values (see below).level
is the current inner tuple's level, starting at zero for the root level.returnData
istrue
if reconstructed data is required for this query; this will only be so if theconfig
function assertedcanReturnData
.allTheSame
is true if the current inner tuple is marked “all-the-same”; in this case all the nodes have the same label (if any) and so either all or none of them match the query (see Section 65.4.3).hasPrefix
is true if the current inner tuple contains a prefix; if so,prefixDatum
is its value.nNodes
is the number of child nodes contained in the inner tuple, andnodeLabels
is an array of their label values, or NULL if the nodes do not have labels.nNodes
must be set to the number of child nodes that need to be visited by the search, andnodeNumbers
must be set to an array of their indexes. If the operator class keeps track of levels, setlevelAdds
to an array of the level increments required when descending to each node to be visited. (Often these increments will be the same for all the nodes, but that's not necessarily so, so an array is used.) If value reconstruction is needed, setreconstructedValues
to an array of the values reconstructed for each child node to be visited; otherwise, leavereconstructedValues
as NULL. The reconstructed values are assumed to be of typespgConfigOut
.leafType
. (However, since the core system will do nothing with them except possibly copy them, it is sufficient for them to have the sametyplen
andtypbyval
properties asleafType
.) If ordered search is performed, setdistances
to an array of distance values according toorderbys
array (nodes with lowest distances will be processed first). Leave it NULL otherwise. If it is desired to pass down additional out-of-band information (“traverse values”) to lower levels of the tree search, settraversalValues
to an array of the appropriate traverse values, one for each child node to be visited; otherwise, leavetraversalValues
as NULL. Note that theinner_consistent
function is responsible for palloc'ing thenodeNumbers
,levelAdds
,distances
,reconstructedValues
, andtraversalValues
arrays in the current memory context. However, any output traverse values pointed to by thetraversalValues
array should be allocated intraversalMemoryContext
. Each traverse value must be a single palloc'd chunk.leaf_consistent
Returns true if a leaf tuple satisfies a query.
The SQL declaration of the function must look like this:
CREATE FUNCTION my_leaf_consistent(internal, internal) RETURNS bool ...
The first argument is a pointer to a
spgLeafConsistentIn
C struct, containing input data for the function. The second argument is a pointer to aspgLeafConsistentOut
C struct, which the function must fill with result data.typedef struct spgLeafConsistentIn { ScanKey scankeys; /* array of operators and comparison values */ ScanKey orderbys; /* array of ordering operators and comparison * values */ int nkeys; /* length of scankeys array */ int norderbys; /* length of orderbys array */ Datum reconstructedValue; /* value reconstructed at parent */ void *traversalValue; /* opclass-specific traverse value */ int level; /* current level (counting from zero) */ bool returnData; /* original data must be returned? */ Datum leafDatum; /* datum in leaf tuple */ } spgLeafConsistentIn; typedef struct spgLeafConsistentOut { Datum leafValue; /* reconstructed original data, if any */ bool recheck; /* set true if operator must be rechecked */ bool recheckDistances; /* set true if distances must be rechecked */ double *distances; /* associated distances */ } spgLeafConsistentOut;
The array
scankeys
, of lengthnkeys
, describes the index search condition(s). These conditions are combined with AND — only index entries that satisfy all of them satisfy the query. (Note thatnkeys
= 0 implies that all index entries satisfy the query.) Usually the consistent function only cares about thesk_strategy
andsk_argument
fields of each array entry, which respectively give the indexable operator and comparison value. In particular it is not necessary to checksk_flags
to see if the comparison value is NULL, because the SP-GiST core code will filter out such conditions. The arrayorderbys
, of lengthnorderbys
, describes the ordering operators in the same manner.reconstructedValue
is the value reconstructed for the parent tuple; it is(Datum) 0
at the root level or if theinner_consistent
function did not provide a value at the parent level.traversalValue
is a pointer to any traverse data passed down from the previous call ofinner_consistent
on the parent index tuple, or NULL at the root level.level
is the current leaf tuple's level, starting at zero for the root level.returnData
istrue
if reconstructed data is required for this query; this will only be so if theconfig
function assertedcanReturnData
.leafDatum
is the key value ofspgConfigOut
.leafType
stored in the current leaf tuple.The function must return
true
if the leaf tuple matches the query, orfalse
if not. In thetrue
case, ifreturnData
istrue
thenleafValue
must be set to the value (of typespgConfigIn
.attType
) originally supplied to be indexed for this leaf tuple. Also,recheck
may be set totrue
if the match is uncertain and so the operator(s) must be re-applied to the actual heap tuple to verify the match. If ordered search is performed, setdistances
to an array of distance values according toorderbys
array. Leave it NULL otherwise. If at least one of returned distances is not exact, setrecheckDistances
to true. In this case, the executor will calculate the exact distances after fetching the tuple from the heap, and will reorder the tuples if needed.
The optional user-defined methods are:
Datum compress(Datum in)
Converts a data item into a format suitable for physical storage in a leaf tuple of the index. It accepts a value of type
spgConfigIn
.attType
and returns a value of typespgConfigOut
.leafType
. The output value must not contain an out-of-line TOAST pointer.Note: the
compress
method is only applied to values to be stored. The consistent methods receive queryscankeys
unchanged, without transformation usingcompress
.options
Defines a set of user-visible parameters that control operator class behavior.
The SQL declaration of the function must look like this:
CREATE OR REPLACE FUNCTION my_options(internal) RETURNS void AS 'MODULE_PATHNAME' LANGUAGE C STRICT;
The function is passed a pointer to a
local_relopts
struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using thePG_HAS_OPCLASS_OPTIONS()
andPG_GET_OPCLASS_OPTIONS()
macros.Since the representation of the key in SP-GiST is flexible, it may depend on user-specified parameters.
All the SP-GiST support methods are normally called in a short-lived memory context; that is, CurrentMemoryContext
will be reset after processing of each tuple. It is therefore not very important to worry about pfree'ing everything you palloc. (The config
method is an exception: it should try to avoid leaking memory. But usually the config
method need do nothing but assign constants into the passed parameter struct.)
If the indexed column is of a collatable data type, the index collation will be passed to all the support methods, using the standard PG_GET_COLLATION()
mechanism.