Thread: pgsql: Permit super-MaxAllocSize allocations with MemoryContextAllocHug
Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). The MaxAllocSize guard is convenient for most callers, because it reduces the need for careful attention to overflow, data type selection, and the SET_VARSIZE() limit. A handful of callers are happy to navigate those hazards in exchange for the ability to allocate a larger chunk. Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX tuples, a factor-of-48 increase. In particular, B-tree index builds can now benefit from much-larger maintenance_work_mem settings. Reviewed by Stephen Frost, Simon Riggs and Jeff Janes. Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/263865a48973767ce8ed7b7788059a38a24a9f37 Modified Files -------------- src/backend/utils/mmgr/aset.c | 5 ++ src/backend/utils/mmgr/mcxt.c | 74 +++++++++++++++++++++++++--- src/backend/utils/sort/tuplesort.c | 91 +++++++++++++++++++---------------- src/backend/utils/sort/tuplestore.c | 77 +++++++++++++++-------------- src/include/utils/memutils.h | 28 ++++++----- src/include/utils/palloc.h | 4 ++ src/include/utils/tuplesort.h | 2 +- 7 files changed, 183 insertions(+), 98 deletions(-)
Re: pgsql: Permit super-MaxAllocSize allocations with MemoryContextAllocHug
From
Andres Freund
Date:
Hi, On 2013-06-27 19:05:22 +0000, Noah Misch wrote: > Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). > > The MaxAllocSize guard is convenient for most callers, because it > reduces the need for careful attention to overflow, data type selection, > and the SET_VARSIZE() limit. A handful of callers are happy to navigate > those hazards in exchange for the ability to allocate a larger chunk. > Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in > tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX > tuples, a factor-of-48 increase. In particular, B-tree index builds can > now benefit from much-larger maintenance_work_mem settings. This commit causes a bunch of warnings like: src/backend/utils/sort/tuplesort.c: In function ‘tuplesort_begin_common’: src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits] #define LACKMEM(state) ((state)->availMem < 0) to be thrown during compilation. And I think it is spot on. Unless you overhaul a good bit of the respective logic making availMem unsigned isn't going to fly. As is, that code will probably be optimized away which isn't good... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, Jul 02, 2013 at 12:08:42PM +0200, Andres Freund wrote: > On 2013-06-27 19:05:22 +0000, Noah Misch wrote: > > Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). > > > > The MaxAllocSize guard is convenient for most callers, because it > > reduces the need for careful attention to overflow, data type selection, > > and the SET_VARSIZE() limit. A handful of callers are happy to navigate > > those hazards in exchange for the ability to allocate a larger chunk. > > Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in > > tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX > > tuples, a factor-of-48 increase. In particular, B-tree index builds can > > now benefit from much-larger maintenance_work_mem settings. > > This commit causes a bunch of warnings like: > > src/backend/utils/sort/tuplesort.c: In function ???tuplesort_begin_common???: > src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false > [-Wtype-limits] > #define LACKMEM(state) ((state)->availMem < 0) > > to be thrown during compilation. And I think it is spot on. Unless you > overhaul a good bit of the respective logic making availMem unsigned > isn't going to fly. True. Will look into it; thanks. -- Noah Misch EnterpriseDB http://www.enterprisedb.com
On Tue, Jul 02, 2013 at 10:33:38AM -0400, Noah Misch wrote: > On Tue, Jul 02, 2013 at 12:08:42PM +0200, Andres Freund wrote: > > On 2013-06-27 19:05:22 +0000, Noah Misch wrote: > > > Permit super-MaxAllocSize allocations with MemoryContextAllocHuge(). > > > > > > The MaxAllocSize guard is convenient for most callers, because it > > > reduces the need for careful attention to overflow, data type selection, > > > and the SET_VARSIZE() limit. A handful of callers are happy to navigate > > > those hazards in exchange for the ability to allocate a larger chunk. > > > Introduce MemoryContextAllocHuge() and repalloc_huge(). Use this in > > > tuplesort.c and tuplestore.c, enabling internal sorts of up to INT_MAX > > > tuples, a factor-of-48 increase. In particular, B-tree index builds can > > > now benefit from much-larger maintenance_work_mem settings. > > > > This commit causes a bunch of warnings like: > > > > src/backend/utils/sort/tuplesort.c: In function ???tuplesort_begin_common???: > > src/backend/utils/sort/tuplesort.c:399:33: warning: comparison of unsigned expression < 0 is always false > > [-Wtype-limits] > > #define LACKMEM(state) ((state)->availMem < 0) > > > > to be thrown during compilation. And I think it is spot on. Unless you > > overhaul a good bit of the respective logic making availMem unsigned > > isn't going to fly. > > True. Will look into it; thanks. In retrospect, I did not need the "long" -> "Size" changes at all. I thought they were necessary to prevent new overflow possibilities on 64-bit Windows, but we already limited work_mem to 2 GiB on that platform to account for its 32-bit "long". That's a limit to be removed rather than perpetuated. Therefore, rather than change back to "long", I've changed to "int64". That's simple to reason about, optimal on 64-bit, and no big loss on 32-bit. Thanks, nm -- Noah Misch EnterpriseDB http://www.enterprisedb.com