Re: [PoC] Improve dead tuple storage for lazy vacuum - Mailing list pgsql-hackers

From Masahiko Sawada
Subject Re: [PoC] Improve dead tuple storage for lazy vacuum
Date
Msg-id CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com
Whole thread Raw
In response to Re: [PoC] Improve dead tuple storage for lazy vacuum  (John Naylor <john.naylor@enterprisedb.com>)
Responses Re: [PoC] Improve dead tuple storage for lazy vacuum
List pgsql-hackers
On Wed, Mar 8, 2023 at 1:40 PM John Naylor <john.naylor@enterprisedb.com> wrote:
>
>

> On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > > 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything
elsethat vacuum does. That would allow other users to avoid that overhead. 
> > > 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm
notsure such a top-down facility exists, and I'm not feeling well enough today to study this further. 
> >
> > I prefer option (1) as it's straight forward. I mentioned a similar
> > idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
> > defined. It might be worth checking if there is visible overhead of
> > tracking chunk memory space. IIRC we've not evaluated it yet.
>
> Ok, let's try this -- I can test and profile later this week.

Thanks!

I've attached the new version patches. I merged improvements and fixes
I did in the v29 patch. 0007 through 0010 are updates from v29. The
main change made in v30 is to make the memory measurement and
RT_MEMORY_USAGE() optional, which is done in 0007 patch. The 0008 and
0009 patches are the updates for tidstore and the vacuum integration
patches. Here are results of quick tests (an average of 3 executions):

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ RT_MEASURE_MEMORY_USAGE:
 mem_allocated | load_ms
---------------+---------
    1996512000 |    3305
(1 row)

* w/o RT_MEASURE_MEMORY_USAGE:
 mem_allocated | load_ms
---------------+---------
             0 |    3258
(1 row)

It seems to be within a noise level but I agree to make it optional.

Apart from the memory measurement stuff, I've done another todo item
on my list; adding min max classes for node3 and node125. I've done
that in 0010 patch, and here is a quick test result:

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ 0000 patch
 mem_allocated | load_ms
---------------+---------
    1268630080 |    3275
(1 row)

* w/o 0000 patch
 mem_allocated | load_ms
---------------+---------
    1996512000 |    3214
(1 row)

That's a good improvement on the memory usage, without a noticeable
performance overhead. FYI CLASS_3_MIN has 1 fanout and is 24 bytes in
size, and CLASS_125_MIN has 61 fanouts and is 768 bytes in size.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachment

pgsql-hackers by date:

Previous
From: Michael Paquier
Date:
Subject: Re: Testing autovacuum wraparound (including failsafe)
Next
From: Michael Paquier
Date:
Subject: Re: Raising the SCRAM iteration count