we are using gin indexes on big tables. these tables happen to have several billion rows.
the index creation fails in ginCombineData in src/backend/access/ginbulk.c because repalloc is limited to 1 gb.
this limitation makes no sense in this context (compare comments in src/include/utils/memutils.h).
To overcome this limitation on tables with lots of rows repalloc_huge is used.
The test suite still succeeds on my machine.
Find the patch attached,
This looks good to me.
One possible issue I see is that if accum.allocatedMemory is only slightly less than maintenance_work_mem just before we decide to repalloc, then doing the repalloc_huge could cause us to exceed maintenance_work_mem. Potentially by a lot. In the worst case (when all the data we've seen during this round of accumulation falls into the same key), we would overrun by a factor of almost 2. Or almost 3, if you count the time during the repalloc when the new data has been allocated but the old data not yet freed. Perhaps the code here should look at the amount of maintenance_work_mem left and grow by less than a factor of 2 if it is too close to going over.
Mitigating that, it won't actually use very much of that memory. Once control passes back at the end of this tuple, the caller will then realize it has overshot the maintenance_work_mem and will flush it all. I think most modern OSes will not have a problems caused by allocated but untouched memory.
This patch doesn't introduce that problem, it just allows it to operate at a higher absolute size. So I'm marking this as ready for committer.
