Thread: Help with PITR WAL Restore and configuration.
Hi All,
We are moving to a new backup solution where we will backup direct to the backup server using a Daily Base backup with pg_start_backup and WAL Log backups every 4 hours.WAL Logs
Daily 4 to 6 times: 60 days retention.
Base Backup
Daily: Mon-Thursday 7 Day Retention
Weekly: Friday 31 Day Retention
Monthly: Last Day of month: 365 Day Retention
So what I am looking for is some clarification on the Base Backups that are 6 months old for example.
Since my WAL Log backups have a 2 month retention I can do PITR from any Base Backup + WAL Logs any day from 2 months back.
However let's say for example I want to restore the Base Backup from 6 months ago, I only have the Base Backups.. Will I be able to recover to that point in time using that Base backup? I have read the PITR documentation however I still struggle to understand this part of it.
Thanks
Jimmy
Let's say that you began this in January. According to your retention plan, you will have:
Base Backups:
--------------
Jan 31, 2014
Feb 28, 2014
Mar 31, 2014
Apr 30, 2014
May 31, 2014
Jun 30, 2014
July 31, 2014
WAL Logs:
Daily WAL logs since Jun 5, 2014
If my understanding is correct, none of your base backups prior to Jun 5, 2014 will be useful for recovering any PITR other than the specific time _of_ the base backup. You cannot replay later WALs on the March backup, for example, as you are missing all of the WAL segments between Mar 31, 2014 and Jun 5, 2014.
So, if I am reading this correctly, the only points which you could recover are:
Jan 31, 2014
Feb 28, 2014
Mar 31, 2014
Apr 30, 2014
May 31, 2014
Jun 30, 2014 + any point in time to your latest WAL backups.
There would be a gap between May 31st and June 5th which would not allow PITR.
In regards to retention we are planning on this.Hi All,We are moving to a new backup solution where we will backup direct to the backup server using a Daily Base backup with pg_start_backup and WAL Log backups every 4 hours.WAL Logs
Daily 4 to 6 times: 60 days retention.
Base BackupDaily: Mon-Thursday 7 Day RetentionWeekly: Friday 31 Day RetentionMonthly: Last Day of month: 365 Day RetentionSo what I am looking for is some clarification on the Base Backups that are 6 months old for example.Since my WAL Log backups have a 2 month retention I can do PITR from any Base Backup + WAL Logs any day from 2 months back.However let's say for example I want to restore the Base Backup from 6 months ago, I only have the Base Backups.. Will I be able to recover to that point in time using that Base backup? I have read the PITR documentation however I still struggle to understand this part of it.ThanksJimmy
You may want to look at the “-x” option for the older backups.
"Includes the required transaction log files (WAL files) in the backup. This will include all transaction logs generated during the backup. If this option is specified, it is possible to start a postmaster directly in the extracted directory without the need to consult the log archive, thus making this a completely standalone backup."
Otherwise you need those wal segments to perform a restore.
-jason
On August 4, 2014 at 11:18:40 AM, Scott Whitney (scott@journyx.com) wrote:
Let's say that you began this in January. According to your retention plan, you will have:
Base Backups:
--------------
Jan 31, 2014
Feb 28, 2014
Mar 31, 2014
Apr 30, 2014
May 31, 2014
Jun 30, 2014
July 31, 2014
WAL Logs:
Daily WAL logs since Jun 5, 2014
If my understanding is correct, none of your base backups prior to Jun 5, 2014 will be useful for recovering any PITR other than the specific time _of_ the base backup. You cannot replay later WALs on the March backup, for example, as you are missing all of the WAL segments between Mar 31, 2014 and Jun 5, 2014.
So, if I am reading this correctly, the only points which you could recover are:
Jan 31, 2014
Feb 28, 2014
Mar 31, 2014
Apr 30, 2014
May 31, 2014
Jun 30, 2014 + any point in time to your latest WAL backups.
There would be a gap between May 31st and June 5th which would not allow PITR.
In regards to retention we are planning on this.Hi All,We are moving to a new backup solution where we will backup direct to the backup server using a Daily Base backup with pg_start_backup and WAL Log backups every 4 hours.WAL Logs
Daily 4 to 6 times: 60 days retention.
Base BackupDaily: Mon-Thursday 7 Day RetentionWeekly: Friday 31 Day RetentionMonthly: Last Day of month: 365 Day RetentionSo what I am looking for is some clarification on the Base Backups that are 6 months old for example.Since my WAL Log backups have a 2 month retention I can do PITR from any Base Backup + WAL Logs any day from 2 months back.However let's say for example I want to restore the Base Backup from 6 months ago, I only have the Base Backups.. Will I be able to recover to that point in time using that Base backup? I have read the PITR documentation however I still struggle to understand this part of it.ThanksJimmy
This transmission contains confidential and privileged information intended solely for the party identified above. If you receive this message in error, you must not use it or convey it to others. Please destroy it immediately and contact the sender at (303) 386-3955 or by return e-mail to the sender.
Hmm, ok that's a concern. Therefore in order for this to work properly (pg_start_bkup method) I need to keep WAL logs for the same retention as the Base backups.
The concern here is that we are moving to a backup solution that grabs only changed blocks so it must be a filesystem level backup to work properly. Actually I do this, run a pg_start_backup then copy the data , then run pg_start_backup. This happens very fast because the solution grabs only the changed blocks since the last backup. Although this is the case each backup is a full backup kinda like synthetic full backup. Using pg_base_backup is going to copy every block every time and it will be really slow for large databses.
So I guess I have a few options.
1) Test pg_basebackup with the "plain format" and hopefully this will write identical data with the exception of the changes each time.Jimmy
On Mon, Aug 4, 2014 at 10:36 AM, Jason Mathis <jmathis@redzonesoftware.com> wrote:
You may want to look at the “-x” option for the older backups."Includes the required transaction log files (WAL files) in the backup. This will include all transaction logs generated during the backup. If this option is specified, it is possible to start a postmaster directly in the extracted directory without the need to consult the log archive, thus making this a completely standalone backup."Otherwise you need those wal segments to perform a restore.-jasonOn August 4, 2014 at 11:18:40 AM, Scott Whitney (scott@journyx.com) wrote:
Let's say that you began this in January. According to your retention plan, you will have:Base Backups:--------------Jan 31, 2014Feb 28, 2014Mar 31, 2014Apr 30, 2014May 31, 2014Jun 30, 2014July 31, 2014WAL Logs:Daily WAL logs since Jun 5, 2014If my understanding is correct, none of your base backups prior to Jun 5, 2014 will be useful for recovering any PITR other than the specific time _of_ the base backup. You cannot replay later WALs on the March backup, for example, as you are missing all of the WAL segments between Mar 31, 2014 and Jun 5, 2014.So, if I am reading this correctly, the only points which you could recover are:Jan 31, 2014Feb 28, 2014Mar 31, 2014Apr 30, 2014May 31, 2014Jun 30, 2014 + any point in time to your latest WAL backups.There would be a gap between May 31st and June 5th which would not allow PITR.In regards to retention we are planning on this.Hi All,We are moving to a new backup solution where we will backup direct to the backup server using a Daily Base backup with pg_start_backup and WAL Log backups every 4 hours.WAL Logs
Daily 4 to 6 times: 60 days retention.
Base BackupDaily: Mon-Thursday 7 Day RetentionWeekly: Friday 31 Day RetentionMonthly: Last Day of month: 365 Day RetentionSo what I am looking for is some clarification on the Base Backups that are 6 months old for example.Since my WAL Log backups have a 2 month retention I can do PITR from any Base Backup + WAL Logs any day from 2 months back.However let's say for example I want to restore the Base Backup from 6 months ago, I only have the Base Backups.. Will I be able to recover to that point in time using that Base backup? I have read the PITR documentation however I still struggle to understand this part of it.ThanksJimmyThis transmission contains confidential and privileged information intended solely for the party identified above. If you receive this message in error, you must not use it or convey it to others. Please destroy it immediately and contact the sender at (303) 386-3955 or by return e-mail to the sender.