Re: pg_combinebackup --incremental - Mailing list pgsql-hackers

Hi,

On Mon, Nov 04, 2024 at 12:53:05PM -0500, Robert Haas wrote:
> Hi,
> 
> When I gave my talk on pg_basebackup at pgconf.eu, people took the
> opportunity to mention various improvements which they would find
> useful. One of those improvements was the ability to combine several
> incremental backups into one. This might be useful if you take very
> frequent incremental backups but want to roll up older ones to reduce
> storage requirements.

+1, I think that could be useful too.

> For example, you could take an incremental
> backup every hour, but then once more than a day has gone by, you
> might roll up those up six at a time into bigger incremental backups
> so that you store 4 backups per day instead of 24 backups per day.
> 
> The attached patch set lets you do this.

Thanks for the patch!

> The idea is that instead of doing this:
> 
> $ pg_combinebackup full incr1 incr2 incr3 incr4 incr5 -o result
> 
> You could instead do this:
> 
> $ pg_combinebackup -i incr1 incr2 incr3 -o incr1-3
> $ rm -rf incr{1,2,3}
> <time passes>
> $ pg_combinebackup full incr1-3 incr5 -o result

Did not test but I guess that one benefit is that the last pg_combinebackup
could be faster than the one that would be needed without the new feature in
place?

> If you're interested in this feature, please give this a try and let
> me know what you find out!

It looks like that it currently does not compile on master, I get things like:

"
reconstruct.c: In function ‘copy_block’:
reconstruct.c:755:50: error: ‘input_filename’ undeclared (first use in this function); did you mean ‘output_filename’?
  755 |                                                  input_filename, output_filename);
      |                                                  ^~~~~~~~~~~~~~
../../../src/include/common/logging.h:152:62: note: in definition of macro ‘pg_fatal’
  152 |                 pg_log_generic(PG_LOG_ERROR, PG_LOG_PRIMARY, __VA_ARGS__); \
      |                                                              ^~~~~~~~~~~
reconstruct.c:755:50: note: each undeclared identifier is reported only once for each function it appears in
  755 |                                                  input_filename, output_filename);
      |                                                  ^~~~~~~~~~~~~~
../../../src/include/common/logging.h:152:62: note: in definition of macro ‘pg_fatal’
  152 |                 pg_log_generic(PG_LOG_ERROR, PG_LOG_PRIMARY, __VA_ARGS__); \
      |                                                              ^~~~~~~~~~~
reconstruct.c:766:25: error: continue statement not within a loop
  766 |                         continue;
      |                         ^~~~~~~~
reconstruct.c:768:28: error: ‘s’ undeclared (first use in this function)
  768 |                 read_block(s, offset, buffer);
      |                            ^
make[3]: *** [../../../src/Makefile.global:961: reconstruct.o] Error 1
"

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com