Re: finding changed blocks using WAL scanning - Mailing list pgsql-hackers

On Sun, Apr 21, 2019 at 06:24:50PM -0400, Robert Haas wrote:
> On Sat, Apr 20, 2019 at 5:54 PM Bruce Momjian <bruce@momjian.us> wrote:
> > Good point.  You mentioned:
> >
> >         It seems better to me to give the files names like
> >         ${TLI}.${STARTLSN}.${ENDLSN}.modblock, e.g.
> >         00000001.0000000168000058.00000001687DBBB8.modblock, so that you can
> >         see exactly which *records* are covered by that segment.
> >
> > but it seems like it should be ${TLI}.${ENDLSN}... (END first) because
> > you would not want to delete the modblock file until you are about to
> > delete the final WAL, not the first WAL, but as you mentioned, it might
> > be ENDLSN-1.
> 
> Hmm.  It seems to me that it is almost universally the convention to
> put the starting point prior to the ending point.  If you are taking a
> biology class, the teacher will not tell you to study chapters six
> through three.

My point is that most WAL archive tools will order and remove files
based on their lexical ordering, so if you put the start first, the file
will normally be removed when it should be kept, e.g., if you have WAL
files like:

    000000010000000000000001
    000000010000000000000002
    000000010000000000000003
    000000010000000000000004
    000000010000000000000005

putting the start first and archiving some wal would lead to:

    000000010000000000000001-000000010000000000000004.modblock
    000000010000000000000003
    000000010000000000000004
    000000010000000000000005

We removed 1 and 2, but kept the modblock file, which looks out of
order. Having the end at the start would have:

    000000010000000000000003
    000000010000000000000004
    000000010000000000000004-000000010000000000000001.modblock
    000000010000000000000005

My point is that you would normally only remove the modblock file when 4
is removed because this modblock files is useful for incremental backups
from base backups that happened between 1 and 4.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +