Re: [HACKERS] On markers of changed data - Mailing list pgsql-hackers

From Greg Stark
Subject Re: [HACKERS] On markers of changed data
Date
Msg-id CAM-w4HME8Kx3saN27CZn-zPw2DWsmc8osXR=8PuJJ0Q2rKU6BQ@mail.gmail.com
Whole thread Raw
In response to Re: [HACKERS] On markers of changed data  (Andrey Borodin <x4mmm@yandex-team.ru>)
Responses Re: [HACKERS] On markers of changed data
List pgsql-hackers
On 8 October 2017 at 08:52, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> 1. Any other marker would be better (It can be WAL scan during archiving, some new LSN-based mechanics* et c.)

The general shape of what I would like to see is some log which lists
where each checkpoint starts and ends and what blocks are modified
since the previous checkpoint. Then to generate an incremental backup
from any point in time to the current you union all the block lists
between them and fetch those blocks. There are other ways of using
this aside from incremental backups on disk too -- you could imagine a
replica that has fallen behind requesting the block lists and then
fetching just those blocks instead of needing to receive and apply all
the wal. Or possibly even making a cost-based decision between the two
depending on which would be faster.

It would also be useful for going in the reverse direction: look up
all the records (or just the last record) that modified a given block.
Instead of having to scan all the wal you would only need to scan the
checkpoint eras that had touched that block.



-- 
greg


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

pgsql-hackers by date:

Previous
From: Petr Jelinek
Date:
Subject: Re: [HACKERS] Help required to debug pg_repack breaking logicalreplication
Next
From: Tomas Vondra
Date:
Subject: [HACKERS] Re: Extended statistics is not working on Vars hidden under aRelabelType