Thread: Re: [WIP]Vertical Clustered Index (columnar store extension) - take2

On Thu, 10 Jul 2025 at 17:51, Peter Smith <smithpb2250@gmail.com> wrote:
> On Thu, Jul 10, 2025 at 4:07 PM Japin Li <japinli@hotmail.com> wrote:
> ...
>> I'm still trying to understand the patches.
>>
>> diff --git a/src/include/storage/itemptr.h b/src/include/storage/itemptr.h
>> index 74b87a9..d97d1c5 100644
>> --- a/src/include/storage/itemptr.h
>> +++ b/src/include/storage/itemptr.h
>> @@ -46,6 +46,9 @@ typedef struct ItemPointerData
>>  #endif
>>  ItemPointerData;
>>
>> +#define SizeOfIptrData \
>> +               (offsetof(ItemPointerData, ip_posid) + sizeof(OffsetNumber))
>> +
>>
>> I've noticed this macro is currently defined within core; however, I found it only
>> used in the VCI extension.
>>
>> Could you clarify the rationale for its inclusion in the core, and whether it's
>> genuinely required there, or if it would be better suited within the extension
>> itself?
>
> Right, this had previously also been reported by Tomas [1].
>
> Upon investigation, I found that this was master code from 10 years
> ago (back when this VCI patch was implemented). The master code has
> moved on since then and removed this macro [2], but this VCI patch did
> not...
>
> I'll try to address this for the next patchset.
>
> ======
> [1] https://www.postgresql.org/message-id/a748aa6b-c7e6-4d02-a590-ab404d590448%40vondra.me
> [2] https://github.com/postgres/postgres/commit/8023b5827fbada6815ce269db4f3373ac77ec7c3
>

My apologies, I wasn't following the full email thread.

--
Regards,
Japin Li

Thank You for working on this. I started going through the README and
tried running simple tests, have few concerns:

1)
I am not able to understand section 4.2 'WOS-to-ROS conversion'. When
whiteout-WOS says 'delete 4', what does that mean? 4 is CRID, TXID?
And when does delete-vector X represents? I did not  get why ColA-2,
ColA-4 and ColB-2, ColB-4 were removed in resultant data? Is the
diagram complete?

2)
We can make the definition consistent at both places as the first one
gives a feeling that rows are marked for deletion in WOS while the
second one says ROS.

Whiteout WOS = Record of WOS rows marked for deletion
Whiteout WOS -- TID records of WOS rows that are marked for deletion on ROS

3)
It is not part of README. But please help me understand the meaning
and usage of this GUC in VCI context:
vci.max_devices: Sets the maximum device number which can be attached.

4)
I was trying my hands on using VCI, I found that on enabling the VCI
path, autovacuum asserts in between. We can reproduce the scenario
using:
create extension vci;
create table tab1( i int, j int, k int);
create index ivci on tab1 using vci (i);
insert into tab1 SELECT generate_series(1,1000),
generate_series(1,1000), generate_series(1,1000);
insert into tab1 SELECT generate_series(1,1000),
generate_series(1,1000), generate_series(1,1000);
SET enable_seqscan TO off;
VACUUM ANALYZE public.tab1;

It asserts here:
TRAP: failed Assert("TransactionIdIsNormal(xmax)"), File: "heapam.c",
Line: 7373, PID: 79834
postgres: autovacuum worker postgres(ExceptionalCondition+0xbb)[0x55fdf8f3ccc6]
postgres: autovacuum worker
postgres(heap_pre_freeze_checks+0x18f)[0x55fdf877e26f]
postgres: autovacuum worker postgres(index_vacuum_cleanup+0x181)[0x55fdf87a1c06]
postgres: autovacuum worker postgres(vac_cleanup_one_index+0x27)[0x55fdf8a0503a]

-------------

Few typos in README:
a) Each VCI indexed column is stored as an internal relations.
--relations --> relation

b) Records are addresses by CRID (Columnar Record ID) instead of by TID.
--addresses->addressed

c) Extents can be found by ID using offsets in a column "meta-data"
internal relation.
-- by ID using offsets? Do you mean 'by using offsets' alone?

d) EXPLAIN ANALYSE -->EXPLAIN ANALYZE


thanks
Shveta

On Fri, Jul 11, 2025 at 1:46 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> Thank You for working on this. I started going through the README and
> tried running simple tests, have few concerns:
>
> 1)
> I am not able to understand section 4.2 'WOS-to-ROS conversion'. When
> whiteout-WOS says 'delete 4', what does that mean? 4 is CRID, TXID?
> And when does delete-vector X represents? I did not  get why ColA-2,
> ColA-4 and ColB-2, ColB-4 were removed in resultant data? Is the
> diagram complete?
>
> 2)
> We can make the definition consistent at both places as the first one
> gives a feeling that rows are marked for deletion in WOS while the
> second one says ROS.
>
> Whiteout WOS = Record of WOS rows marked for deletion
> Whiteout WOS -- TID records of WOS rows that are marked for deletion on ROS
>
> 3)
> It is not part of README. But please help me understand the meaning
> and usage of this GUC in VCI context:
> vci.max_devices: Sets the maximum device number which can be attached.
>
> 4)
> I was trying my hands on using VCI, I found that on enabling the VCI
> path, autovacuum asserts in between. We can reproduce the scenario
> using:
> create extension vci;
> create table tab1( i int, j int, k int);
> create index ivci on tab1 using vci (i);
> insert into tab1 SELECT generate_series(1,1000),
> generate_series(1,1000), generate_series(1,1000);
> insert into tab1 SELECT generate_series(1,1000),
> generate_series(1,1000), generate_series(1,1000);
> SET enable_seqscan TO off;
> VACUUM ANALYZE public.tab1;
>
> It asserts here:
> TRAP: failed Assert("TransactionIdIsNormal(xmax)"), File: "heapam.c",
> Line: 7373, PID: 79834
> postgres: autovacuum worker postgres(ExceptionalCondition+0xbb)[0x55fdf8f3ccc6]
> postgres: autovacuum worker
> postgres(heap_pre_freeze_checks+0x18f)[0x55fdf877e26f]
> postgres: autovacuum worker postgres(index_vacuum_cleanup+0x181)[0x55fdf87a1c06]
> postgres: autovacuum worker postgres(vac_cleanup_one_index+0x27)[0x55fdf8a0503a]
>
> -------------
>
> Few typos in README:
> a) Each VCI indexed column is stored as an internal relations.
> --relations --> relation
>
> b) Records are addresses by CRID (Columnar Record ID) instead of by TID.
> --addresses->addressed
>
> c) Extents can be found by ID using offsets in a column "meta-data"
> internal relation.
> -- by ID using offsets? Do you mean 'by using offsets' alone?
>
> d) EXPLAIN ANALYSE -->EXPLAIN ANALYZE
>
>

Hi Shveta.

Thanks for your README questions and typo reports. I will try to
address those soon.

Regarding your reported VACUUM exception, I believe this is the same
issue that was previously reported by Japin Li [1].

======
[1]
https://www.postgresql.org/message-id/SY8P300MB0442BEC3F5CF432F0121ACC4B642A%40SY8P300MB0442.AUSP300.PROD.OUTLOOK.COM

Kind Regards,
Peter Smith.
Fujitsu Australia