Thread: Named restore points
Hi, Here is a patch that implements "named restore points". It allows DBAs to specify an exact point to which they can recover but that point will have a name, so they have a better control of when they want to stop recovery (ie: DBA's won't depend of remember specific times, dates and such). This adds a new function: pg_create_restore_point(text) (i'm not wedded with the name so if someone wants to suggest something better, that's fine with me), a new xlog record and a new recovery_target parameter in recovery.conf -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL
Attachment
Jaime Casanova <jaime@2ndquadrant.com> writes: > Here is a patch that implements "named restore points". > It allows DBAs to specify an exact point to which they can recover > but that point will have a name, so they have a better control of when > they want to stop recovery (ie: DBA's won't depend of remember > specific times, dates and such). > This adds a new function: pg_create_restore_point(text) (i'm not > wedded with the name so if someone wants to suggest something better, > that's fine with me), a new xlog record and a new recovery_target > parameter in recovery.conf This seems like it's a lot of mechanism for an awfully small use-case. How often are people actually going to have the foresight to know that "right now" is when they would want to restore to later? And is it really any easier to use a label for that than a timestamp? You're still going to need to keep track of which label means what. regards, tom lane
Em 14-01-2011 17:41, Jaime Casanova escreveu: > Here is a patch that implements "named restore points". > Nice feature. I only read the provided documentation and it seems inconsistent to allow name, time, and xid at recovery_target_name because (i) someone could name the recovery point as '1234567' (xid) or '2011-01-14' (I use this format a lot) and (ii) if the suffix name is *_name* it shouldn't allow xid and time. IMHO, recovery_target_name should allow only names. -- Euler Taveira de Oliveira http://www.timbira.com/
On Fri, Jan 14, 2011 at 5:42 PM, Euler Taveira de Oliveira <euler@timbira.com> wrote: > Em 14-01-2011 17:41, Jaime Casanova escreveu: >> >> Here is a patch that implements "named restore points". >> > Nice feature. I only read the provided documentation and it seems > inconsistent to allow name, time, and xid at recovery_target_name it only allow names, but those names could be anything -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL
On Fri, Jan 14, 2011 at 3:41 PM, Jaime Casanova <jaime@2ndquadrant.com> wrote: > Here is a patch that implements "named restore points". > > It allows DBAs to specify an exact point to which they can recover > but that point will have a name, so they have a better control of when > they want to stop recovery (ie: DBA's won't depend of remember > specific times, dates and such). > > This adds a new function: pg_create_restore_point(text) (i'm not > wedded with the name so if someone wants to suggest something better, > that's fine with me), a new xlog record and a new recovery_target > parameter in recovery.conf Neat. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, 2011-01-14 at 17:18 -0500, Tom Lane wrote: > Jaime Casanova <jaime@2ndquadrant.com> writes: > > Here is a patch that implements "named restore points". > > > It allows DBAs to specify an exact point to which they can recover > > but that point will have a name, so they have a better control of when > > they want to stop recovery (ie: DBA's won't depend of remember > > specific times, dates and such). > > > This adds a new function: pg_create_restore_point(text) (i'm not > > wedded with the name so if someone wants to suggest something better, > > that's fine with me), a new xlog record and a new recovery_target > > parameter in recovery.conf > > This seems like it's a lot of mechanism for an awfully small use-case. > How often are people actually going to have the foresight to know that > "right now" is when they would want to restore to later? And is it > really any easier to use a label for that than a timestamp? You're > still going to need to keep track of which label means what. I think its the other way around. In order to know what time to restore to, you have to keep an external list of times when interesting things happened. This gives you a way of putting that metadata into the log stream so everything you need is in one place. You can put a restore point in before or after any major activity, so you can restore the database if that fails. e.g. 'daily backup 2001/1/11', 'reference data update 2011/2/5', 'pg_upgrade', etc.. -- Simon Riggs http://www.2ndQuadrant.com/books/PostgreSQL Development, 24x7 Support, Training and Services
Em 14-01-2011 19:50, Jaime Casanova escreveu: > On Fri, Jan 14, 2011 at 5:42 PM, Euler Taveira de Oliveira > <euler@timbira.com> wrote: >> Em 14-01-2011 17:41, Jaime Casanova escreveu: >>> >>> Here is a patch that implements "named restore points". >>> >> Nice feature. I only read the provided documentation and it seems >> inconsistent to allow name, time, and xid at recovery_target_name > > it only allow names, but those names could be anything > OK. I will review your patch at the beginning of the week. -- Euler Taveira de Oliveira http://www.timbira.com/
On Fri, Jan 14, 2011 at 8:33 PM, Euler Taveira de Oliveira <euler@timbira.com> wrote: > > OK. I will review your patch at the beginning of the week. > thanks -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL
On Fri, Jan 14, 2011 at 8:33 PM, Euler Taveira de Oliveira <euler@timbira.com> wrote: > OK. I will review your patch at the beginning of the week. Euler, are you still planning to review this? We're running out of time. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Em 14-01-2011 17:41, Jaime Casanova escreveu: > Here is a patch that implements "named restore points". > Sorry, I was swamped with work. :( Your patch no longer applied so I rebased it and slightly modified it. Review is below... + The default is to recover to the end of the WAL log. + The precise stopping point is also influenced by + <xref linkend="recovery-target-inclusive">. + </para> This isn't valid. recovery_target_name are not influenced by recovery_target_inclusive. Sentence removed. + static char recoveryStopNamedRestorePoint[MAXFNAMELEN]; Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking at code...] It seems to be used for backup label too so it is not so inappropriate. + typedef struct xl_named_restore_points + { + TimestampTz xtime; + char name[MAXFNAMELEN]; + } xl_named_restore_points; + I prefixed those struct members so it won't get confused elsewhere. + else if (recoveryTarget == RECOVERY_TARGET_NAME) + snprintf(buffer, sizeof(buffer), + "%s%u\t%s\t%s named restore point %s\n", + (srcfd < 0) ? "" : "\n", + parentTLI, + xlogfname, + recoveryStopAfter ? "after" : "before", + recoveryStopNamedRestorePoint); It doesn't matter if it is after or before the restore point. After/Before only make sense when we're dealing with transaction or time. Removed. else if (strcmp(item->name, "recovery_target_xid") == 0) { + /* + * if recovery_target_name specified, then this overrides + * recovery_target_xid + */ + if (recoveryTarget == RECOVERY_TARGET_NAME) + continue; + IMHO the right recovery precedence is xid -> name -> time. If you're specifying xid that's because you know what you are doing. Name takes precedence over time because it is easier to remember a name than a time. I implemented this order in the updated patch. + recoveryTargetName = pstrdup(item->value); I also added a check for long names. + if ((record->xl_rmid == RM_XLOG_ID) && (record_info == XLOG_RESTORE_POINT)) + couldStop = true; + + if (!couldStop) + return false; + I reworked this code path because it seems confusing. + recordNamedRestorePoint = (xl_named_restore_points *) XLogRecGetData(record); + recordXtime = recordNamedRestorePoint->xtime; Why don't you store the named restore point here too? You will need it a few lines below. + char name[MAXFNAMELEN]; + + memcpy(&xlrec, rec, sizeof(xl_named_restore_points)); + strncpy(name, xlrec.name, MAXFNAMELEN); Is it really necessary? I removed it. + Datum + pg_create_restore_point(PG_FUNCTION_ARGS) + { You should have added a check for long restore point names. Added in the updated patch. + ereport(NOTICE, + (errmsg("WAL archiving is not enabled; you must ensure that WAL segments are copied through other means for restore points to be usefull for you"))); + Sentence was rewritten as "WAL archiving is not enabled; you must ensure that WAL segments are copied through other means to recover up to named restore point". Finally, this is a nice feature iif we have a way to know what named restore points are available. DBAs need to take note of this list (that is not good) and the lazy ones will have a hard time to recover the right name (possibly with a xlog dump tool). So how could we store this information? Perhaps a file in $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL location). Also it must have a way to transmit the restore_label when we add another restore point. I didn't implement this part (Jaime?) and it seems as important as the new xlog record type that is in the patch. It seems complicate but I don't have ideas. Anyone? The restore point names could be obtained by querying a function (say, pg_restore_point_names or pg_restore_point_list). Someone could argue that this feature could be reached if we store label and WAL location in a file (say restore_label). This mechanism doesn't need a new WAL record but the downside is that if we lost restore_label we are dead. I'm not in favor of this approach because it seems too fragile. I will mark this patch waiting on author because of those open issues. This patch needs to bump catalog version because of the new function. I'm not sure if the new record type requires bumping the xlog magic number. I'm attaching the updated patch and two scripts that I used to play with the patch. -- Euler Taveira de Oliveira http://www.timbira.com/
Attachment
On Tue, Feb 1, 2011 at 10:02 AM, Euler Taveira de Oliveira <euler@timbira.com> wrote: > Em 14-01-2011 17:41, Jaime Casanova escreveu: >> >> Here is a patch that implements "named restore points". >> > Sorry, I was swamped with work. :( > > Your patch no longer applied so I rebased it and slightly modified it. > Review is below... > Hi, Thanks for the review, i've been without internet connection for 4 days so i haven't seen the review until now... > + The default is to recover to the end of the WAL log. > + The precise stopping point is also influenced by > + <xref linkend="recovery-target-inclusive">. > + </para> > > This isn't valid. recovery_target_name are not influenced by > recovery_target_inclusive. Sentence removed. > good point! docs are boring so i was in automatic mode ;) > + static char recoveryStopNamedRestorePoint[MAXFNAMELEN]; > > Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking > at code...] It seems to be used for backup label too so it is not so > inappropriate. > right, i used it because it is used for backup label > > + else if (recoveryTarget == RECOVERY_TARGET_NAME) > + snprintf(buffer, sizeof(buffer), > + "%s%u\t%s\t%s named restore point %s\n", > + (srcfd < 0) ? "" : "\n", > + parentTLI, > + xlogfname, > + recoveryStopAfter ? "after" : "before", > + recoveryStopNamedRestorePoint); > > It doesn't matter if it is after or before the restore point. After/Before > only make sense when we're dealing with transaction or time. Removed. > you're right > else if (strcmp(item->name, "recovery_target_xid") == 0) > { > + /* > + * if recovery_target_name specified, then this > overrides > + * recovery_target_xid > + */ > + if (recoveryTarget == RECOVERY_TARGET_NAME) > + continue; > + > > IMHO the right recovery precedence is xid -> name -> time. If you're > specifying xid that's because you know what you are doing. Name takes > precedence over time because it is easier to remember a name than a time. I > implemented this order in the updated patch. > actually i was expecting to hear opinions about this and i agree with you > + recoveryTargetName = pstrdup(item->value); > > I also added a check for long names. > ok > + if ((record->xl_rmid == RM_XLOG_ID) && (record_info == > XLOG_RESTORE_POINT)) > + couldStop = true; > + > + if (!couldStop) > + return false; > + > > I reworked this code path because it seems confusing. > it is... it was the result of debugging an stupid error on my side... > + recordNamedRestorePoint = (xl_named_restore_points *) > XLogRecGetData(record); > + recordXtime = recordNamedRestorePoint->xtime; > > Why don't you store the named restore point here too? You will need it a few > lines below. > don't remember, will see > > + Datum > + pg_create_restore_point(PG_FUNCTION_ARGS) > + { > > You should have added a check for long restore point names. Added in the > updated patch. > ok > + ereport(NOTICE, > + (errmsg("WAL archiving is not enabled; you must ensure > that WAL segments are copied through other means for restore points to be > usefull for you"))); > + > > Sentence was rewritten as "WAL archiving is not enabled; you must ensure > that WAL segments are copied through other means to recover up to named > restore point". > sounds better, thanks > Finally, this is a nice feature iif we have a way to know what named restore > points are available. DBAs need to take note of this list (that is not good) > and the lazy ones will have a hard time to recover the right name (possibly > with a xlog dump tool). > > So how could we store this information? Perhaps a file in > $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL > location). Also it must have a way to transmit the restore_label when we add > another restore point. I didn't implement this part (Jaime?) and it seems as > important as the new xlog record type that is in the patch. It seems > complicate but I don't have ideas. Anyone? The restore point names could be > obtained by querying a function (say, pg_restore_point_names or > pg_restore_point_list). > IMHO, probably the best answer is a tool to retrieve that info... the problem is that a "restore_label" file should be closely attached to the WAL segment where the named restore point is... and a sql function won't say anything about named restore points that are in archived WAL segments... > > I will mark this patch waiting on author because of those open issues. > > I'm attaching the updated patch and two scripts that I used to play with the > patch. > ok, i will see you're reviewed version later today -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL
On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova <jaime@2ndquadrant.com> wrote: > ok, i will see you're reviewed version later today This patch is still marked as "Needs Review" in the CommitFest application, but I'm thinking perhaps it should be changed to Ready for Committer? Are there any open issues? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Mon, Feb 7, 2011 at 10:59 PM, Robert Haas <robertmhaas@gmail.com> wrote: > On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova <jaime@2ndquadrant.com> wrote: >> ok, i will see you're reviewed version later today > > This patch is still marked as "Needs Review" in the CommitFest > application, but I'm thinking perhaps it should be changed to Ready > for Committer? Are there any open issues? > only things i can found are: > + static char recoveryStopNamedRestorePoint[MAXFNAMELEN]; > > Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking > at code...] It seems to be used for backup label too so it is not so > inappropriate. > which is just a question about if MAXFNAMELEN is the right length to use and > Finally, this is a nice feature iif we have a way to know what named restore > points are available. DBAs need to take note of this list (that is not good) > and the lazy ones will have a hard time to recover the right name (possibly > with a xlog dump tool). > > So how could we store this information? Perhaps a file in > $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL > location). Also it must have a way to transmit the restore_label when we add > another restore point. I didn't implement this part (Jaime?) and it seems as > important as the new xlog record type that is in the patch. It seems > complicate but I don't have ideas. Anyone? The restore point names could be > obtained by querying a function (say, pg_restore_point_names or > pg_restore_point_list). > i still think this should be a separate tool or a dba written list, the reason beign that with sql function we were not able track restore points in archived segments... if you like i can try to build a simple tool for this but don't think that is a showstopper, even without that the feature is useful IMHO at least -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte y capacitación de PostgreSQL
On Tue, Feb 8, 2011 at 2:05 AM, Jaime Casanova <jaime@2ndquadrant.com> wrote: >> Finally, this is a nice feature iif we have a way to know what named restore >> points are available. DBAs need to take note of this list (that is not good) >> and the lazy ones will have a hard time to recover the right name (possibly >> with a xlog dump tool). >> >> So how could we store this information? Perhaps a file in >> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL >> location). Also it must have a way to transmit the restore_label when we add >> another restore point. I didn't implement this part (Jaime?) and it seems as >> important as the new xlog record type that is in the patch. It seems >> complicate but I don't have ideas. Anyone? The restore point names could be >> obtained by querying a function (say, pg_restore_point_names or >> pg_restore_point_list). >> > > i still think this should be a separate tool or a dba written list, I agree. Keeping track of where you've set named restore points is not going to be a problem with a simple solution. Which restore points are available is going to depend on which base backup you restored and what WAL files you stuffed into pg_xlog. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, 2011-02-08 at 08:05 -0500, Robert Haas wrote: > On Tue, Feb 8, 2011 at 2:05 AM, Jaime Casanova <jaime@2ndquadrant.com> wrote: > >> Finally, this is a nice feature iif we have a way to know what named restore > >> points are available. DBAs need to take note of this list (that is not good) > >> and the lazy ones will have a hard time to recover the right name (possibly > >> with a xlog dump tool). > >> > >> So how could we store this information? Perhaps a file in > >> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL > >> location). Also it must have a way to transmit the restore_label when we add > >> another restore point. I didn't implement this part (Jaime?) and it seems as > >> important as the new xlog record type that is in the patch. It seems > >> complicate but I don't have ideas. Anyone? The restore point names could be > >> obtained by querying a function (say, pg_restore_point_names or > >> pg_restore_point_list). > >> > > > > i still think this should be a separate tool or a dba written list, > > I agree. Keeping track of where you've set named restore points is > not going to be a problem with a simple solution. Which restore > points are available is going to depend on which base backup you > restored and what WAL files you stuffed into pg_xlog. Yeah agreed. No need for restore_label -- Simon Riggs http://www.2ndQuadrant.com/books/PostgreSQL Development, 24x7 Support, Training and Services
On Fri, 2011-02-04 at 21:15 -0500, Jaime Casanova wrote: > > > > + else if (recoveryTarget == RECOVERY_TARGET_NAME) > > + snprintf(buffer, sizeof(buffer), > > + "%s%u\t%s\t%s named restore point % > s\n", > > + (srcfd < 0) ? "" : "\n", > > + parentTLI, > > + xlogfname, > > + recoveryStopAfter ? "after" : > "before", > > + recoveryStopNamedRestorePoint); > > > > It doesn't matter if it is after or before the restore point. > After/Before > > only make sense when we're dealing with transaction or time. > Removed. > > > > you're right Not sure I understand the comment "only make sense when we're dealing with transaction or time." Why? At present, I think the ability to stop before/after a named restore point should be put back. -- Simon Riggs http://www.2ndQuadrant.com/books/PostgreSQL Development, 24x7 Support, Training and Services
Em 08-02-2011 11:05, Simon Riggs escreveu: > On Fri, 2011-02-04 at 21:15 -0500, Jaime Casanova wrote: >>> >>> + else if (recoveryTarget == RECOVERY_TARGET_NAME) >>> + snprintf(buffer, sizeof(buffer), >>> + "%s%u\t%s\t%s named restore point % >> s\n", >>> + (srcfd< 0) ? "" : "\n", >>> + parentTLI, >>> + xlogfname, >>> + recoveryStopAfter ? "after" : >> "before", >>> + recoveryStopNamedRestorePoint); >>> >>> It doesn't matter if it is after or before the restore point. >> After/Before >>> only make sense when we're dealing with transaction or time. >> Removed. >>> >> >> you're right > > Not sure I understand the comment "only make sense when we're dealing > with transaction or time." Why? > Because named restore point is a noop xlog record; besides, transaction and time involves xlog records that contain data. -- Euler Taveira de Oliveira http://www.timbira.com/
On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote: > > Not sure I understand the comment "only make sense when we're dealing > > with transaction or time." Why? > > > Because named restore point is a noop xlog record; besides, transaction and > time involves xlog records that contain data. Thank you. How obvious! -- Simon Riggs http://www.2ndQuadrant.com/books/PostgreSQL Development, 24x7 Support, Training and Services
On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote: > Because named restore point is a noop xlog record; besides, transaction and > time involves xlog records that contain data. Committed. Thanks for the patch and the review. I changed the patch to require wal_level > minimal, rather than archive_mode = on. -- Simon Riggs http://www.2ndQuadrant.com/books/PostgreSQL Development, 24x7 Support, Training and Services
On 8 February 2011 19:53, Simon Riggs <simon@2ndquadrant.com> wrote: > On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote: > >> Because named restore point is a noop xlog record; besides, transaction and >> time involves xlog records that contain data. > > Committed. Thanks for the patch and the review. > > I changed the patch to require wal_level > minimal, rather than > archive_mode = on. This could do with a bit more documentation about usage. Below the Backup Control Functions table (http://developer.postgresql.org/pgdocs/postgres/functions-admin.html#FUNCTIONS-ADMIN-BACKUP-TABLE), each function has a paragraph detailing what it does. Also, I notice you can easily write over a label. The case I'm thinking of is someone in psql creating a named restore point, then later on, they go in again, accidentally cursor up and select the previous statement and create it again. Would this mean that the previous label is lost, or would it be the case that any subsequent duplicate labels would have no effect unless the WAL files with the original label in were consumed? In either case, a note in the docs about this would be useful. And I don't see these label creations getting logged either. Could we output that to the log because at least then users can grep the directory for labels, and, in most cases, the time they occurred? -- Thom Brown Twitter: @darkixion IRC (freenode): dark_ixion Registered Linux user: #516935
On Wed, Feb 9, 2011 at 4:53 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > Committed. Thanks for the patch and the review. - * We also track the timestamp of the latest applied COMMIT/ABORT record - * in XLogCtl->recoveryLastXTime, for logging purposes. + * We also track the timestamp of the latest applied COMMIT/ABORT/RESTORE POINT + * record in XLogCtl->recoveryLastXTime, for logging purposes. Tracking the timestamp of the restore point record in recoveryLastXTime messes up pg_last_xact_replay_timestamp which uses recoveryLastXTime. The timestamp of the restore point is wrongly returned as that of the latest transaction, by the function. As far as I read the patch, I don't think that it's necessary to track the timestamp of the restore point. The attached patch changes the code so that it doesn't track the timestamp of the restore point. + if (strlen(restore_name_str) >= MAXFNAMELEN) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("value too long for restore point"))); I think that logging the maximum length of the name is useful as follows: ERROR: value too long for restore point (max 63 characters) So the attached patch also changes the log message that way. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
Attachment
Em 08-02-2011 17:35, Thom Brown escreveu: > This could do with a bit more documentation about usage. Below the > Backup Control Functions table > (http://developer.postgresql.org/pgdocs/postgres/functions-admin.html#FUNCTIONS-ADMIN-BACKUP-TABLE), > each function has a paragraph detailing what it does. > I forgot to check it. > Also, I notice you can easily write over a label. The case I'm > thinking of is someone in psql creating a named restore point, then > later on, they go in again, accidentally cursor up and select the > previous statement and create it again. Would this mean that the > previous label is lost, or would it be the case that any subsequent > duplicate labels would have no effect unless the WAL files with the > original label in were consumed? In either case, a note in the docs > about this would be useful. > This is a limitation that I pointed out [1] but people decided to postpone named restore point management. The first one is used as restore point. I added it in the attached patch. > And I don't see these label creations getting logged either. Could we > output that to the log because at least then users can grep the > directory for labels, and, in most cases, the time they occurred? > Good point. I included location instead of time; time is already supplied by log file. The following patch implements the Thom's suggestions. [1] http://archives.postgresql.org/message-id/4D48209C.7050109@timbira.com -- Euler Taveira de Oliveira http://www.timbira.com/
Attachment
On Thu, Feb 24, 2011 at 10:28 AM, Euler Taveira de Oliveira <euler@timbira.com> wrote: > The following patch implements the Thom's suggestions. > > [1] http://archives.postgresql.org/message-id/4D48209C.7050109@timbira.com Committed with some additional wordsmithing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company