From 359d06adb9924266bc9022295ae2c68964703f70 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 14 Dec 2019 11:36:28 +1300 Subject: [PATCH] Rotate instead of shifting hash join batch number. Our algorithm for choosing batch numbers turned out not to work effectively for multi-billion key hash tables. We would try to use more hash bits than we have, and effectively concentrate all tuples into a smaller number of batches than we intended. While ideally we should switch to wider hashes, for now, change the algorithm to one that effectively gives up bits from the bucket number instead, leaving some buckets empty. Author: Thomas Munro Reported-by: James Coleman Discussion: https://postgr.es/m/16104-dc11ed911f1ab9df%40postgresql.org --- src/backend/executor/nodeHash.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c index d6f4eda097..5bcdc5b5d5 100644 --- a/src/backend/executor/nodeHash.c +++ b/src/backend/executor/nodeHash.c @@ -1877,7 +1877,7 @@ ExecHashGetHashValue(HashJoinTable hashtable, * chains), and must only cause the batch number to remain the same or * increase. Our algorithm is * bucketno = hashvalue MOD nbuckets - * batchno = (hashvalue DIV nbuckets) MOD nbatch + * batchno = ROT(hashvalue, log2_nbuckets) MOD nbatch * where nbuckets and nbatch are both expected to be powers of 2, so we can * do the computations by shifting and masking. (This assumes that all hash * functions are good about randomizing all their output bits, else we are @@ -1900,11 +1900,12 @@ ExecHashGetBucketAndBatch(HashJoinTable hashtable, uint32 nbuckets = (uint32) hashtable->nbuckets; uint32 nbatch = (uint32) hashtable->nbatch; +#define rotate(v, n) (((v) >> (n)) | ((v) << (((sizeof(v) * CHAR_BIT) - (n))))) + if (nbatch > 1) { - /* we can do MOD by masking, DIV by shifting */ *bucketno = hashvalue & (nbuckets - 1); - *batchno = (hashvalue >> hashtable->log2_nbuckets) & (nbatch - 1); + *batchno = rotate(hashvalue, hashtable->log2_nbuckets) & (nbatch - 1); } else { -- 2.23.0