Re: Vacuum: allow usage of more than 1GB of work mem - Mailing list pgsql-hackers
From | Andrew Dunstan |
---|---|
Subject | Re: Vacuum: allow usage of more than 1GB of work mem |
Date | |
Msg-id | 6b52b65c-e694-c768-8358-5b4ffa0c37bb@2ndQuadrant.com Whole thread Raw |
In response to | Re: Vacuum: allow usage of more than 1GB of work mem (Claudio Freire <klaussfreire@gmail.com>) |
Responses |
Re: Vacuum: allow usage of more than 1GB of work mem
(Claudio Freire <klaussfreire@gmail.com>)
|
List | pgsql-hackers |
On 07/16/2018 10:34 AM, Claudio Freire wrote: > On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan > <andrew.dunstan@2ndquadrant.com> wrote: >> >> >> On 07/13/2018 09:44 AM, Heikki Linnakangas wrote: >>> On 13/07/18 01:39, Andrew Dunstan wrote: >>>> On 07/12/2018 06:34 PM, Alvaro Herrera wrote: >>>>> On 2018-Jul-12, Andrew Dunstan wrote: >>>>> >>>>>> I fully understand. I think this needs to go back to "Waiting on >>>>>> Author". >>>>> Why? Heikki's patch applies fine and passes the regression tests. >>>> Well, I understood Claudio was going to do some more work (see >>>> upthread). >>> Claudio raised a good point, that doing small pallocs leads to >>> fragmentation, and in particular, it might mean that we can't give >>> back the memory to the OS. The default glibc malloc() implementation >>> has a threshold of 4 or 32 MB or something like that - allocations >>> larger than the threshold are mmap()'d, and can always be returned to >>> the OS. I think a simple solution to that is to allocate larger >>> chunks, something like 32-64 MB at a time, and carve out the >>> allocations for the nodes from those chunks. That's pretty >>> straightforward, because we don't need to worry about freeing the >>> nodes in retail. Keep track of the current half-filled chunk, and >>> allocate a new one when it fills up. >> >> Google seems to suggest the default threshold is much lower, like 128K. >> Still, making larger allocations seems sensible. Are you going to work >> on that? > Below a few MB the threshold is dynamic, and if a block bigger than > 128K but smaller than the higher threshold (32-64MB IIRC) is freed, > the dynamic threshold is set to the size of the freed block. > > See M_MMAP_MAX and M_MMAP_THRESHOLD in the man page for mallopt[1] > > So I'd suggest allocating blocks bigger than M_MMAP_MAX. > > [1] http://man7.org/linux/man-pages/man3/mallopt.3.html That page says: M_MMAP_MAX This parameter specifies the maximum number of allocation requests that may be simultaneously serviced using mmap(2). This parameter exists because some systems have a limited number of internal tables for use by mmap(2), and using more than a few of them may degrade performance. The default value is 65,536, a value which has no special significance and which serves only as a safeguard. Setting this parameter to 0 disables the use of mmap(2) for servicing large allocation requests. I'm confused about the relevance. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
pgsql-hackers by date: