RE: Recovery performance of standby for multiple concurrenttruncates on large tables - Mailing list pgsql-hackers

From Jamison, Kirk
Subject RE: Recovery performance of standby for multiple concurrenttruncates on large tables
Date
Msg-id D09B13F772D2274BB348A310EE3027C631D7D3@g01jpexmbkw24
Whole thread Raw
In response to Re: Recovery performance of standby for multiple concurrenttruncates on large tables  ('Andres Freund' <andres@anarazel.de>)
Responses Re: Recovery performance of standby for multiple concurrent truncateson large tables  (Robert Haas <robertmhaas@gmail.com>)
Re: Recovery performance of standby for multiple concurrenttruncates on large tables  ('Andres Freund' <andres@anarazel.de>)
List pgsql-hackers
Hi Andres,

> I think this is a case where the potential work arounds are complex enough to use significant resources to get right,
andare likely to make properly fixing the issue harder. I'm willing to comment on proposals that claim not to be
problmaticin those regards, but I have *SERIOUS* doubts they exist.
 

Alright. But I'd still try and ask your thoughts about it (below).
The proposed design touches the buffer invalidation during recovery process of standby server.

The question was about "how" to remember those buffers that contain truncate/drop tables, right?

1. Because the multiple scans of the whole shared buffer per concurrent truncate/drop table was the cause of the
time-consumingbehavior, DURING the failover process while WAL is being applied, we temporary delay the scanning and
invalidatingof shared buffers. At the same time, we remember the relations/relfilenodes (of dropped/truncated tables)
byadding them in a hash table called "skip list". 
 
2. After WAL is applied, the checkpoint(or bg writer) scans the shared buffer only ONCE, compare the pages against the
skiplist, and invalidates the relevant pages. After deleting the relevant pages on the shared memory, it will not be
writtenback to the disk.
 

Assuming the theory works, this design will only affect the behavior of checkpointer (or maybe bg writer) during
recoveryprocess / failover. Any feedback, thoughts?
 

BTW, are there any updates whether the community will push through anytime soon regarding the buffer mapping
implementationyou mentioned?
 


Regards,
Kirk Jamison



pgsql-hackers by date:

Previous
From: Melanie Plageman
Date:
Subject: Re: Making "COPY partitioned_table FROM" faster
Next
From: Kyotaro HORIGUCHI
Date:
Subject: Re: Usability fail with psql's \dp command