Re: BUG #14963: Number of wal files are keep on increasing - Mailing list pgsql-bugs

From Michael Paquier
Subject Re: BUG #14963: Number of wal files are keep on increasing
Date
Msg-id CAB7nPqR_9gUz3_p-xdn_+PnvuTP-Zo=T09AcZzLfvMa3+gTPaw@mail.gmail.com
Whole thread Raw
In response to Re: BUG #14963: Number of wal files are keep on increasing  (Stephen Frost <sfrost@snowman.net>)
Responses Re: BUG #14963: Number of wal files are keep on increasing  (Stephen Frost <sfrost@snowman.net>)
List pgsql-bugs
On Thu, Dec 14, 2017 at 10:35 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
>> Raghavendra Rao J S V wrote:
>> > We are taking the backup of database everyday using pg_basebackup utility. We
>> > have implemented a pg_basebackup utility through shell script.This will be
>> > invoked everyday at 2AM and takes the backup of database.
>>
>> This is a bad idea.  If you don't have a very firm grasp of how to write
>> a backup system, it's better not to write it yourself, because there are
>> many possible errors you can make that can lead you to end up with
>> backups that look ok but are actually invalid.
>
> Using pg_basebackup is *not* a bad idea, it's a good tool and works just
> fine if used properly for a lot of environments.  The biggest concern
> with pg_basebackup is making sure that the WAL is streamed with the
> backup and kept (using --xlog-method=stream) and that you're checking
> the results of the command to make sure it didn't error out.

What I understand here is Alvaro telling that a shell script mimicking
pg_basebackup behavior is a bad idea, as Raghavendra seems to mean.
And that's right to not encourage people to do that. pg_basebackup is
a robust and excellent tool, maintained with the core code and we've
made things better with the latest releases, like improving. It is
used everyday in my internal company stuff and we don't regret it, so
use it :)

Note also a couple of things which have changed in Postgres 10 to ease
pg_basebackup's use:
1) --xlog-method (actually renamed to --wal-method) uses the stream
method by default.
2) By default, again, a temporary replication slot is used so as you
don't need to worry about WAL segment rotation caused by segments.
3) Default parameters of postgresql.conf and pg_hba.conf have been
changed to ease the use of those tools, like max_wal_senders and
max_replication_slots with non-0 defaults, replication connections
allowed.

> Of course, with any backup solution, it's essential that backups be
> fully restored and tested to make sure that they work.

"Consider yourself as not having a backup unless you have tested that
you are able to restore it".

> There are certainly other tools out there for doing backups of PG with
> their own pros and cons and while I wouldn't recommend that people try
> to write their own backup software, using pg_basebackup isn't bad and we
> shouldn't be telling people that.

pg_basebackup has hundreds man-hours behind, if not more. Any custom
and quickly-written script will never match its quality.
-- 
Michael


pgsql-bugs by date:

Previous
From: Raghavendra Rao J S V
Date:
Subject: Re: BUG #14963: Number of wal files are keep on increasing
Next
From: Stephen Frost
Date:
Subject: Re: BUG #14963: Number of wal files are keep on increasing