RE: Recovery performance of standby for multiple concurrenttruncates on large tables - Mailing list pgsql-hackers

From Jamison, Kirk
Subject RE: Recovery performance of standby for multiple concurrenttruncates on large tables
Date
Msg-id D09B13F772D2274BB348A310EE3027C6300333@g01jpexmbkw24
Whole thread Raw
In response to Re: Recovery performance of standby for multiple concurrenttruncates on large tables  (Andres Freund <andres@anarazel.de>)
Responses Re: Recovery performance of standby for multiple concurrenttruncates on large tables
List pgsql-hackers

Hi,

Thank you for your replies.

 

On Tue, July 10, 2018 4:15 PM, Andres Freund wrote:

>I think you'd run into a lot of very hairy details with this approach. Consider what happens if client processes need fresh buffers and need to write out a victim buffer. You'll need to know that the relevant buffer is actually invalid. Thus the knowledge about the "delayed" drops would need to be in shared buffers and scanned on every dirty buffer writeout.

 

Yes, you're right. There are problems associated with checkpoint as you pointed out. I just thought of touching the involved process on writing dirty pages to the disk, which are both the functions of checkpoint and background writer.

> How about using the background writer for this? ...

> ...

And now that I think about it, the suggestion of Ants above on "background writer" would seem to work better, as bg writer is more active and continuously write dirty pages but checkpointer only does it in interval. This seems to be a more viable solution and I'd appreciate advice. I'll explore around this idea, and if possible, I'd like to develop a solution for the next commitfest.

 

> I personally think we should rather just work towards a ordered buffer mapping implementation.

 

I understand that the main problem lies on shared_buffers scanning and that a buffer mapping implementation is underway(?) for PG12. I am not sure if the community has arrived at a consensus for that long-term fix. Would the community also welcome any minor/smaller-scale improvements (just for this particular problem on wal recovery for truncate table)? If yes, then I'd like to work on a possible solution.

 

Regards,

Kirk Jamison

pgsql-hackers by date:

Previous
From: Asim R P
Date:
Subject: Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket
Next
From: Michael Paquier
Date:
Subject: Re: print_path is missing GatherMerge and CustomScan support