Thread: Load distributed checkpoint patch

This is a patch for load distributed checkpoint discussed in
http://archives.postgresql.org/pgsql-hackers/2006-12/msg00337.php

Only write() calls are smoothed, fsync() are not.
Existing checkpoint method is called "immediate checkpoint" in the patch,
and the new method called "asynchronous checkpoint".

I'm working on more robust control of checkpoint progress. WAL are written
a lot just after checkpoints because of full-page-writes and decreases
gradually. Linear control seems to be not enough in some cases.

Comments, suggestions and testing are appreciated.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

ITAGAKI Takahiro wrote:
> This is a patch for load distributed checkpoint discussed in
> http://archives.postgresql.org/pgsql-hackers/2006-12/msg00337.php
>
> Only write() calls are smoothed, fsync() are not.
> Existing checkpoint method is called "immediate checkpoint" in the patch,
> and the new method called "asynchronous checkpoint".
>
> I'm working on more robust control of checkpoint progress. WAL are written
> a lot just after checkpoints because of full-page-writes and decreases
> gradually. Linear control seems to be not enough in some cases.
>
> Comments, suggestions and testing are appreciated.

OK, if I understand correctly, instead of doing a buffer scan, write(),
and fsync(), and recyle the WAL files at checkpoint time, you delay the
scan/write part with the some delay.  Do you use the same delay
autovacuum uses?  As I remember, often the checkpoint is caused because
we are using the last WAL file.  Doesn't this delay the creation of new
WAL files by renaming the old ones to higher numbers (we can't rename
them until the checkpoint is complete)?

--
  Bruce Momjian   bruce@momjian.us
  EnterpriseDB    http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

The attached is a revised patch for smoothing checkpoints. The added 3 GUC
parameters can be used to control sleeps in each stage during checkpoints.
The another is an approach to replace fsync() for fine control.

1. checkpoint_write_duration (default=0, in seconds)
        Sets the duration of write() phase in checkpoints.
2. checkpoint_nap_duration (default=0, in seconds)
        Sets the duration between write() and fsync() phases in checkpoints.
3. checkpoint_sync_duration (default=0, in seconds)
        Sets the duration of fsync() phase in checkpoints.
4. checkpoint_sync_size (default=0, in KB)
        Sets the synchronization unit of data files in checkpoints.

I'll send the detail to -hackers.
Any comments and testing reports will be appreciated.


> This is a patch for load distributed checkpoint discussed in
> http://archives.postgresql.org/pgsql-hackers/2006-12/msg00337.php
>
> Only write() calls are smoothed, fsync() are not.
> Existing checkpoint method is called "immediate checkpoint" in the patch,
> and the new method called "asynchronous checkpoint".

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center