On Thu, 2020-03-19 at 15:33 -0400, Robert Haas wrote:
> On Thu, Mar 19, 2020 at 3:27 PM Jeff Davis <pgsql@j-davis.com> wrote:
> > I think omitting the tail of the current block is an unqualified
> > improvement for the purpose of obeying work_mem, regardless of the
> > OS.
> > The block sizes keep doubling up to 8MB, and it doesn't make a lot
> > of
> > sense to count that last large mostly-empty block against work_mem.
>
> Well, again, my point is that whether or not it counts depends on
> your
> system's overcommit behavior. Depending on how you have the
> configured, or what your OS likes to do, it may in reality count or
> not count. Or so I believe. Am I wrong?
I don't believe it should not be counted for the purposes of work_mem.
Let's say that the OS eagerly allocates it, then what is the algorithm
supposed to do in response? It can either:
1. just accept that all of the space is used, even though it's
potentially as low as 50% used, and spill earlier than may be
necessary; or
2. find a way to measure the free space, and somehow predict whether
that space will be reused the next time a group is added to the hash
table
It just doesn't seem reasonable for me for the algorithm to change its
behavior based on these large block allocations. It may be valuable
information for other purposes (like tuning your OS, or tracking down
memory issues), though.
Regards,
Jeff Davis
