Hi,
On 2025-03-23 01:45:35 -0700, Masahiko Sawada wrote:
> Another idea is that parallel workers don't exit phase 1 until it
> consumes all pinned buffers in the queue, even if the memory usage of
> TidStore exceeds the limit.
Yes, that seems a quite reasonable approach to me.
> It would need to add new functionality to the read stream to disable the
> look-ahead reading.
Couldn't your next block callback simply return InvalidBlockNumber once close
to the memory limit?
> Since we could use much memory while processing these buffers, exceeding the
> memory limit, we can trigger this mode when the memory usage of TidStore
> reaches 70% of the limit or so.
It wouldn't be that much memory, would it? A few 10s-100s of buffers don't
increase the size of a TidStore that much? Using 10 parallel vacuum with a
m_w_m of 1MB doesn't make sense, we'd constantly start/stop workers.
> On the other hand, it means that we would not use the streaming read for the
> blocks in this mode, which is not efficient.
I don't follow - why wouldn't you be using streaming read?
Greetings,
Andres Freund
