Thread: [ADMIN] pg_backrest no longer backing up
Hello,
Sometime in December in 2016 we ran out of space for our backup directory, which we increased promptly. However, it appears that backrest stopped taking backups since the incident.
Versions of things:
pg_backrest 0.75
postgresql 9.4
ubuntu 14.04 (This is the backup server, not master)
Couple things I noticed:
postgres@pgdedicated:/backup/backrest/log$ cat prod-9.4-backup-20170308.log
-------------------PROCESS START-------------------
2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
2017-03-08 01:30:02.835 T00 INFO: last backup label = 20161204-025729F, version = 0.75
2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with label "pg_backrest backup started 2017-03-08 01:30:02": backup will begin after the requested immediate checkpoint completes
2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or exception, 0 threads stopped
postgres@pgdedicated:/backup/backrest/log$ ls -lah /backup/backrest/backup/prod-9.4/
total 44K
drwxrwxr-x 11 postgres postgres 4.0K Dec 22 14:02 .
drwxrwxr-x 3 postgres postgres 4.0K Mar 15 2016 ..
drwxrwxr-x 3 postgres postgres 4.0K Nov 20 03:03 20161120-030305F
drwxrwxr-x 3 postgres postgres 4.0K Nov 27 02:53 20161127-025355F
drwxrwxr-x 3 postgres postgres 4.0K Dec 3 02:24 20161127-025355F_20161203-022437D
drwxrwxr-x 3 postgres postgres 4.0K Dec 4 02:57 20161204-025729F
drwxrwxr-x 3 postgres postgres 4.0K Dec 5 02:24 20161204-025729F_20161205-022427D
drwxrwxr-x 3 postgres postgres 4.0K Dec 6 02:25 20161204-025729F_20161206-022512D
drwxrwxr-x 3 postgres postgres 4.0K Dec 7 02:28 20161204-025729F_20161207-022814D
drwxrwxr-x 3 postgres postgres 4.0K Dec 8 02:27 20161204-025729F_20161208-022730D
drwxrwxr-x 3 postgres postgres 4.0K Dec 9 02:28 20161204-025729F_20161209-022855D
lrwxrwxrwx 1 postgres postgres 33 Dec 9 02:28 latest -> 20161204-025729F_20161209-022855D
postgres@pgdedicated:/backup/backrest/log$ cat /etc/cron.d/backrest
# Vacuum every Sunday before Full backup
#05 00 * * 0 postgres vacuumdb -qaz
# Full backup every Sunday
#30 01 * * 0 postgres /usr/lib/backrest/bin/pg_backrest.pl --stanza=prod-9.4 --type=full backup
# Differential backup all other days
#30 01 * * 1,2,3,4,5,6 postgres /usr/lib/backrest/bin/pg_backrest.pl --stanza=prod-9.4 --type=diff backup
postgres@pgdedicated:/backup/backrest/log$ cat /etc/pg_backrest.conf
[global:general]
repo-path=/backup/backrest
[global:backup]
start-fast=y
[global:expire]
retention-full=6 # This was originally 8, dialed it down to 6 when ran out of space
retention-diff=6
retention-archive-type=full
retention-archive=6 # This was originally 8, dialed it down to 6 when ran out of space
[prod-9.4]
db-path=/var/lib/postgresql/9.4/replica
Question: Not sure when the backrest cron was commented out, but how is it still attempting to do backups without this setup? I'm assuming it's being invoked some other way?
Also, any idea what the issue is with the error?
Hi Aldo, On 3/8/17 11:48 AM, Aldo Sarmiento wrote: > Sometime in December in 2016 we ran out of space for our backup > directory, which we increased promptly. However, it appears that > backrest stopped taking backups since the incident. > > Versions of things: > > pg_backrest 0.75 This is an incredibly old, pre-1.0 version of pgBackRest. I suggest you upgrade to a current version. 1.X will not be compatible with your current repo so it will need to be a fresh install. > *postgres@pgdedicated:/backup/backrest/log$ cat > prod-9.4-backup-20170308.log* > -------------------PROCESS START------------------- > 2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff > 2017-03-08 01:30:02.835 T00 INFO: last backup label = > 20161204-025729F, version = 0.75 > 2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with > label "pg_backrest backup started 2017-03-08 01:30:02": backup will > begin after the requested immediate checkpoint completes > 2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or > exception, 0 threads stopped This could be any number of things and there's not enough information given here to even begin to guess. > Question: Not sure when the backrest cron was commented out, but how is > it still attempting to do backups without this setup? I'm assuming it's > being invoked some other way? I'm guessing it's being invoked in some other way, perhaps the postgres user's crontab? > Also, any idea what the issue is with the error? Again, there's not enough information given for me to form an opinion. Regards, -- -David david@pgmasters.net
Hi David,
Actually got it figured out. Main thing is that I should clarify that the aforementioned crontab was irrelevant since I was looking at the replica.
Once logged into master, the crontab is intact and uncommented. The problem was that a backup failed and pg_backrest was failing to start.
I tried adding auto-stop=true to conf, but it didn't help. In the end I just did pg_stop_backup() and things are running smoothly now.
As for the upgrade, yes that's a need we have. Just trying to see if we can migrate our current backups to work with the new version so we don't lose ability for PITR.
Thanks!
Actually got it figured out. Main thing is that I should clarify that the aforementioned crontab was irrelevant since I was looking at the replica.
Once logged into master, the crontab is intact and uncommented. The problem was that a backup failed and pg_backrest was failing to start.
I tried adding auto-stop=true to conf, but it didn't help. In the end I just did pg_stop_backup() and things are running smoothly now.
As for the upgrade, yes that's a need we have. Just trying to see if we can migrate our current backups to work with the new version so we don't lose ability for PITR.
Thanks!
On Wed, Mar 8, 2017 at 2:18 PM David Steele <david@pgmasters.net> wrote:
Hi Aldo,
On 3/8/17 11:48 AM, Aldo Sarmiento wrote:
> Sometime in December in 2016 we ran out of space for our backup
> directory, which we increased promptly. However, it appears that
> backrest stopped taking backups since the incident.
>
> Versions of things:
>
> pg_backrest 0.75
This is an incredibly old, pre-1.0 version of pgBackRest. I suggest you
upgrade to a current version. 1.X will not be compatible with your
current repo so it will need to be a fresh install.
> *postgres@pgdedicated:/backup/backrest/log$ cat
> prod-9.4-backup-20170308.log*
> -------------------PROCESS START-------------------
> 2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
> 2017-03-08 01:30:02.835 T00 INFO: last backup label =
> 20161204-025729F, version = 0.75
> 2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with
> label "pg_backrest backup started 2017-03-08 01:30:02": backup will
> begin after the requested immediate checkpoint completes
> 2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or
> exception, 0 threads stopped
This could be any number of things and there's not enough information
given here to even begin to guess.
> Question: Not sure when the backrest cron was commented out, but how is
> it still attempting to do backups without this setup? I'm assuming it's
> being invoked some other way?
I'm guessing it's being invoked in some other way, perhaps the postgres
user's crontab?
> Also, any idea what the issue is with the error?
Again, there's not enough information given for me to form an opinion.
Regards,
--
-David
david@pgmasters.net
Hi Aldo, On 3/8/17 5:53 PM, Aldo Sarmiento wrote: > > Actually got it figured out. Main thing is that I should clarify that > the aforementioned crontab was irrelevant since I was looking at the > replica. > > Once logged into master, the crontab is intact and uncommented. That makes sense. > The > problem was that a backup failed and pg_backrest was failing to start. > > I tried adding auto-stop=true to conf, but it didn't help. In the end I > just did pg_stop_backup() and things are running smoothly now. The option is actually stop-auto: http://www.pgbackrest.org/user-guide.html#backup/option-stop-auto But it wasn't added until 0.80 so wouldn't work for you anyway. > As for the upgrade, yes that's a need we have. Just trying to see if we > can migrate our current backups to work with the new version so we don't > lose ability for PITR. It might be possible but would require specialized knowledge and custom code. We decided to make a clean break at 1.0 and require users to create a new repository due to the number of changes in the format. Regards, -- -David david@pgmasters.net