Nope, that's not how it works. It's the array of batches that gets
sliced, not the batches themselves.
It does slightly increase the amount of data we need to shuffle between
the temp files, because we can't write the data directly to batches in
"future" slices. But that amplification is capped to ~2.2x (compared to
the ~1.4x in master) - I've shared some measurements in [1].
[1] https://www.postgresql.org/message-id/20190428141901.5dsbge2ka3rxmpk6%40development
Cool, I misunderstood. I looked at the code again today, and, at the email
thread where you measured "amplification".
In terms of how many times you write each tuple, is it accurate to say that a
tuple can now be spilled three times (in the worst case) whereas, before, it
could be spilled only twice?
1 - when building the inner side hashtable, tuple is spilled to a "slice" file
2 - (assuming the number of batches was increased) during execution, when a
tuple belonging to a later slice's spill file is found, it is re-spilled to that
slice's spill file
3 - during execution, when reading from its slice file, it is re-spilled (again)
to its batch's spill file
Is it correct that the max number of BufFile structs you will have is equal to
the number of slices + number of batches in a slice
because that is the max number of open BufFiles you would have at a time?
By the way, applying v4 patch on master, in an assert build, I am tripping some
asserts -- starting with
Assert(!file->readOnly);
in BufFileWrite
One thing I was a little confused by was the nbatch_inmemory member of the
hashtable. The comment in ExecChooseHashTableSize says that it is determining
the number of batches we can fit in memory. I thought that the problem was the
amount of space taken up by the BufFile data structure itself--which is related
to the number of open BufFiles you need at a time. This comment in
ExecChooseHashTableSize makes it sound like you are talking about fitting more
than one batch of tuples into memory at a time. I was under the impression that
you could only fit one batch of tuples in memory at a time.
So, I was stepping through the code with work_mem set to the lower bound, and in
ExecHashIncreaseNumBatches, I got confused.
hashtable->nbatch_inmemory was 2 for me, thus, nbatch_tmp was 2
so, I didn't meet this condition
if (nbatch_tmp > hashtable->nbatch_inmemory)
since I just set nbatch_tmp using hashtable->nbatch_inmemory
So, I didn't increase the number of slices, which is what I was expecting.
What happens when hashtable->nbatch_inmemory is equal to nbatch_tmp?
--
