On Wed, Apr 16, 2025 at 12:31 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> More or less, yeah, just put the whole ReadStream object in shared
> memory, pin an LWLock on it and call it a parallel-aware or shared
> ReadStream. But how do you make the locking not terrible?
>
> My "work stealing" brain dump was imagining a way to achieve the same
> net effect, except NOT have to acquire an exclusive lock for every
> buffer you pull out of the stream. I was speculating that we could
> achieve zero locking for most of the stream without any cache line
> ping pong, but a cunning read barrier scheme could detect when you've
> been flipped into a slower coordination mode by another backend and
> need to turn on some locking and fight over the last handful of
> buffers. And I was also observing that if you can figure out to make
> it general and reusable enough, we have more unsolved problems like
> this in unrelated parallel query code not even involving streams.
> It's a tiny more approachable subset of the old "data buffered in
> other workers" problem, as I think you called it once.
Maybe the work stealing stuff can hide inside the ReadStream? e.g. a
ParallelReadStream is really one ReadStream per participant, each with
a separate lock. Mostly you only touch your own, but if necessary you
can poke your fingers into other people's ReadStreams.
--
Robert Haas
EDB: http://www.enterprisedb.com
