I didn't propose too different solution. There is only one difference - sharing DSM for smaller data. It is similar to using usual shared memory.
Does this mean implementing some sort of allocator on top of the shared memory segment? If so, how are you going to prevent fragmentation?
yes, simple memory allocator is necessary in this case. But it should be really simple - you can allocate only fixed size blocks - 10KB, 100KB and 1MB from separate buffers. So the fragmentation is not possible.
Maybe we're talking about completely different ideas, but I don't see how fixing the block helps to fight fragmentation, in particular. Can you sketch a simple example? E.g. 400 backends share the common segment and all of them want to publish a plan of ~1kb, for example. Now what?
Probably only few backends will be data requesters and few backends will be requested for data. The size of shared data will be typically less than 10KB, with few exceptions 100KB, and few exceptions 10MB. So for 400 backends you need 400*10KB+100*100KB+20*1MB = 34MB. You can have three independent buffers for this sizes. If some process require 5KB then you returns 10KB (it is good enough) from first buffer. This simple mechanism can coverage 90% of usage. for other 10% you can create new DSM. Because you are working with fixed size blocks and you don't divide the size of block, then the fragmentation isn't possible. This model allow very fast allocation, very fast deallocation.
It is not terribly effective, but 34MB is better than 400 DSM or than 400MB (max size for any backend).
Our super long queries are terribly long > 1MB, but we use a few clients (less than 2x CPU cores)
