> This seems like a rather low limit that will definitely affect
> folks in the field.
I am not a heavy user of pg_repack, just curious, why do we think
105 million is a very small number?
> An "easy" fix for this might be to mark that allocation as
> using palloc_huge() somehow.
The request that fails is the comboCids array in GetComboCommandId(),
not the hash table. 8 bytes times 209,715,200 entries is the
1677721600 in Radim's error. I do not see a limit in the hash table.
I tried repalloc_array_extended() there. With 110 million replayed updates
master fails with that error, and the patched build completes with
correct data. It only changes runs that fail today. From reading the
code, the next limits are the int array size at about 1.7 billion
entries, and then 2^32 commands.
It does not lower the memory use. I measured 48 bytes for each
change, so the limit today also caps this at about 5 GB. Without it,
1 billion changes need about 50 GB, and a host with less gets the
OOM killer, not an ERROR.
Experimental patch attached.
Thanks,
Shihao
