Re: [PERFORM] Backup taking long time !!! - Mailing list pgsql-performance

From Stephen Frost
Subject Re: [PERFORM] Backup taking long time !!!
Date
Msg-id 20170120134054.GH18360@tamriel.snowman.net
Whole thread Raw
In response to Re: [PERFORM] Backup taking long time !!!  (Vladimir Borodin <root@simply.name>)
List pgsql-performance
Vladimir,

* Vladimir Borodin (root@simply.name) wrote:
> > 20 янв. 2017 г., в 15:22, Stephen Frost <sfrost@snowman.net> написал(а):
> >> This process can be automatized by some applications like barman
> >> http://www.pgbarman.org/
> >
> > Last I checked, barman is still single-threaded.
> >
> > If the database is large enough that you need multi-process backup, I'd
> > suggest looking at pgbackrest- http://www.pgbackrest.org.
> >
> > pgbackrest has parallel backup, incremental/differential/full backup
> > support, supports compression, CRC checking, and a whole ton of other
> > good stuff.
>
> Increments in pgbackrest are done on file level which is not really efficient. We have done parallelism, compression
andpage-level increments (9.3+) in barman fork [1], but unfortunately guys from 2ndquadrant-it don’t hurry to work on
it.

We're looking at page-level incremental backup in pgbackrest also.  For
larger systems, we've not heard too much complaining about it being
file-based though, which is why it hasn't been a priority.  Of course,
the OP is on 9.1 too, so.

As for your fork, well, I can't say I really blame the barman folks for
being cautious- that's usually a good thing in your backup software. :)

I'm curious how you're handling compressed page-level incremental
backups though.  I looked through barman-incr and it wasn't obvious to
me what was going wrt how the incrementals are stored, are they ending
up as sparse files, or are you actually copying/overwriting the prior
file in the backup repository?  Apologies, python isn't my first
language, but the lack of any comment anywhere in that file doesn't
really help.

> And actually it would be much better to do a good backup and recovery manager part of the core postgres.

Sure, but that's not going to happen for 9.1, or even 9.6, and I doubt
PG10 is going to suddenly get parallel base-backup with compression.

I've been discussing ways to improve the situation with Magnus and we do
have some ideas about it, but that's really an independent effort as
we're still going to need a tool for released versions of PG.

Thanks!

Stephen

Attachment

pgsql-performance by date:

Previous
From: Vladimir Borodin
Date:
Subject: Re: [PERFORM] Backup taking long time !!!
Next
From: Vladimir Borodin
Date:
Subject: Re: [PERFORM] Backup taking long time !!!