Thread: Interesting post-mortem on a near disaster with git
Over the weekend, KDE came within a gnat's eyelash of losing *all* their authoritative git repos, despite having seemingly-extensive redundancy. Read about it here: http://jefferai.org/2013/03/24/too-perfect-a-mirror/ We should think about protecting our own repo a bit better, especially after the recent unpleasantness with a bogus forced update. The idea of having clones that are deliberately a day or two behind seems attractive ... regards, tom lane
On Sun, Mar 24, 2013 at 11:52:17AM -0400, Tom Lane wrote: > Over the weekend, KDE came within a gnat's eyelash of losing *all* > their authoritative git repos, despite having seemingly-extensive > redundancy. Read about it here: > http://jefferai.org/2013/03/24/too-perfect-a-mirror/ > > We should think about protecting our own repo a bit better, especially > after the recent unpleasantness with a bogus forced update. The idea > of having clones that are deliberately a day or two behind seems > attractive ... I think the lesson here is that a mirror is not a backup. RAID, ZFS, and version control are all not backups. Taking a tarball of the entire repository and storing it on a different machine would solve just about any problem you can think of in this area. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > He who writes carelessly confesses thereby at the very outset that he does > not attach much importance to his own thoughts. -- Arthur Schopenhauer
On 03/24/2013 05:08 PM, Martijn van Oosterhout wrote: > On Sun, Mar 24, 2013 at 11:52:17AM -0400, Tom Lane wrote: >> Over the weekend, KDE came within a gnat's eyelash of losing *all* >> their authoritative git repos, despite having seemingly-extensive >> redundancy. Read about it here: >> http://jefferai.org/2013/03/24/too-perfect-a-mirror/ >> >> We should think about protecting our own repo a bit better, especially >> after the recent unpleasantness with a bogus forced update. The idea >> of having clones that are deliberately a day or two behind seems >> attractive ... > > I think the lesson here is that a mirror is not a backup. RAID, ZFS, > and version control are all not backups. > > Taking a tarball of the entire repository and storing it on a different > machine would solve just about any problem you can think of in this > area. fwiw - the sysadmin team has file-level backups of all pginfra hosts (two backups/day, one per day for a week and a full per week for 4 weeks of history). Stefan
On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Michael
Over the weekend, KDE came within a gnat's eyelash of losing *all*
their authoritative git repos, despite having seemingly-extensive
redundancy. Read about it here:
http://jefferai.org/2013/03/24/too-perfect-a-mirror/
It is really great that KDE people are actually sharing this experience. This is really profitable for other projects as well as individuals.
And thanks for sharing it here.
And thanks for sharing it here.
We should think about protecting our own repo a bit better, especially
after the recent unpleasantness with a bogus forced update. The idea
of having clones that are deliberately a day or two behind seems
attractive ...
Just an idea here: why not adding a new subdomain in postgresql.org for mirrors of the official GIT repository
similar to the buildfarm?
People registered in this service could publish themselves mirrors and decide by themselves the delay their
clone keeps with the parent repo. The scripts used by each mirror maintainer (for backup, sync repo with
a given delay) could be centralized in a way similar to buildfarm code so as everybody in the community could
use it and publish it if they want.
Also, the mirrors published should be maintained by people that are well-known inside the community,
and who would not add extra commits which would make the mirror out-of-sync with the parent repo.
Such an idea is perhaps too much if the point is to maintain 2-3 mirrors of the parent repo, but gives
enough transparency to actually know where the mirrors are and what is the sync delay maintained.
--
similar to the buildfarm?
People registered in this service could publish themselves mirrors and decide by themselves the delay their
clone keeps with the parent repo. The scripts used by each mirror maintainer (for backup, sync repo with
a given delay) could be centralized in a way similar to buildfarm code so as everybody in the community could
use it and publish it if they want.
Also, the mirrors published should be maintained by people that are well-known inside the community,
and who would not add extra commits which would make the mirror out-of-sync with the parent repo.
Such an idea is perhaps too much if the point is to maintain 2-3 mirrors of the parent repo, but gives
enough transparency to actually know where the mirrors are and what is the sync delay maintained.
--
On 03/24/2013 06:06 PM, Michael Paquier wrote: > On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane <tgl@sss.pgh.pa.us > <mailto:tgl@sss.pgh.pa.us>> wrote: > > Over the weekend, KDE came within a gnat's eyelash of losing *all* > their authoritative git repos, despite having seemingly-extensive > redundancy. Read about it here: > http://jefferai.org/2013/03/24/too-perfect-a-mirror/ > > It is really great that KDE people are actually sharing this > experience. This is really profitable for other projects as well as > individuals. > And thanks for sharing it here. > > > We should think about protecting our own repo a bit better, especially > after the recent unpleasantness with a bogus forced update. The idea > of having clones that are deliberately a day or two behind seems > attractive ... > > Just an idea here: why not adding a new subdomain in postgresql.org > <http://postgresql.org> for mirrors of the official GIT repository > similar to the buildfarm? > People registered in this service could publish themselves mirrors and > decide by themselves the delay their > clone keeps with the parent repo. The scripts used by each mirror > maintainer (for backup, sync repo with > a given delay) could be centralized in a way similar to buildfarm code > so as everybody in the community could > use it and publish it if they want. > > Also, the mirrors published should be maintained by people that are > well-known inside the community, > and who would not add extra commits which would make the mirror > out-of-sync with the parent repo. > > Such an idea is perhaps too much if the point is to maintain 2-3 > mirrors of the parent repo, but gives > enough transparency to actually know where the mirrors are and what is > the sync delay maintained. This strikes me as being overkill. The sysadmins seem to have it covered. Back when we used CVS for quite a few years I kept 7 day rolling snapshots of the CVS repo, against just such a difficulty as this. But we seem to be much better organized with infrastructure these days so I haven't done that for a long time. cheers andrew
On 03/24/2013 11:22 PM, Andrew Dunstan wrote: > > On 03/24/2013 06:06 PM, Michael Paquier wrote: >> On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane <tgl@sss.pgh.pa.us >> <mailto:tgl@sss.pgh.pa.us>> wrote: >> >> Over the weekend, KDE came within a gnat's eyelash of losing *all* >> their authoritative git repos, despite having seemingly-extensive >> redundancy. Read about it here: >> http://jefferai.org/2013/03/24/too-perfect-a-mirror/ >> >> It is really great that KDE people are actually sharing this >> experience. This is really profitable for other projects as well as >> individuals. >> And thanks for sharing it here. >> >> >> We should think about protecting our own repo a bit better, >> especially >> after the recent unpleasantness with a bogus forced update. The idea >> of having clones that are deliberately a day or two behind seems >> attractive ... >> >> Just an idea here: why not adding a new subdomain in postgresql.org >> <http://postgresql.org> for mirrors of the official GIT repository >> similar to the buildfarm? >> People registered in this service could publish themselves mirrors and >> decide by themselves the delay their >> clone keeps with the parent repo. The scripts used by each mirror >> maintainer (for backup, sync repo with >> a given delay) could be centralized in a way similar to buildfarm code >> so as everybody in the community could >> use it and publish it if they want. >> >> Also, the mirrors published should be maintained by people that are >> well-known inside the community, >> and who would not add extra commits which would make the mirror >> out-of-sync with the parent repo. >> >> Such an idea is perhaps too much if the point is to maintain 2-3 >> mirrors of the parent repo, but gives >> enough transparency to actually know where the mirrors are and what is >> the sync delay maintained. > > > > This strikes me as being overkill. The sysadmins seem to have it covered. > > Back when we used CVS for quite a few years I kept 7 day rolling > snapshots of the CVS repo, against just such a difficulty as this. But > we seem to be much better organized with infrastructure these days so I > haven't done that for a long time. well there is always room for improvement(and for learning from others) - but I agree that this proposal seems way overkill. If people think we should keep online "delayed" mirrors we certainly have the resources to do that on our own if we want though... Stefan
On Mon, Mar 25, 2013 at 7:07 PM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote: > On 03/24/2013 11:22 PM, Andrew Dunstan wrote: >> >> On 03/24/2013 06:06 PM, Michael Paquier wrote: >>> On Mon, Mar 25, 2013 at 12:52 AM, Tom Lane <tgl@sss.pgh.pa.us >>> <mailto:tgl@sss.pgh.pa.us>> wrote: >>> >>> Over the weekend, KDE came within a gnat's eyelash of losing *all* >>> their authoritative git repos, despite having seemingly-extensive >>> redundancy. Read about it here: >>> http://jefferai.org/2013/03/24/too-perfect-a-mirror/ >>> >>> It is really great that KDE people are actually sharing this >>> experience. This is really profitable for other projects as well as >>> individuals. >>> And thanks for sharing it here. >>> >>> >>> We should think about protecting our own repo a bit better, >>> especially >>> after the recent unpleasantness with a bogus forced update. The idea >>> of having clones that are deliberately a day or two behind seems >>> attractive ... >>> >>> Just an idea here: why not adding a new subdomain in postgresql.org >>> <http://postgresql.org> for mirrors of the official GIT repository >>> similar to the buildfarm? >>> People registered in this service could publish themselves mirrors and >>> decide by themselves the delay their >>> clone keeps with the parent repo. The scripts used by each mirror >>> maintainer (for backup, sync repo with >>> a given delay) could be centralized in a way similar to buildfarm code >>> so as everybody in the community could >>> use it and publish it if they want. >>> >>> Also, the mirrors published should be maintained by people that are >>> well-known inside the community, >>> and who would not add extra commits which would make the mirror >>> out-of-sync with the parent repo. >>> >>> Such an idea is perhaps too much if the point is to maintain 2-3 >>> mirrors of the parent repo, but gives >>> enough transparency to actually know where the mirrors are and what is >>> the sync delay maintained. >> >> >> >> This strikes me as being overkill. The sysadmins seem to have it covered. >> >> Back when we used CVS for quite a few years I kept 7 day rolling >> snapshots of the CVS repo, against just such a difficulty as this. But >> we seem to be much better organized with infrastructure these days so I >> haven't done that for a long time. > > well there is always room for improvement(and for learning from others) > - but I agree that this proposal seems way overkill. If people think we > should keep online "delayed" mirrors we certainly have the resources to > do that on our own if we want though... Yeah, definitely. It's also interesting to note that one of the things they do is to "stop using mirrored clones". The fact that we *don't* use mirrored clones for our anon repository is exectly how we caught the issue caused by the invalid push that Kevin did a short while ago... -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Mon, Mar 25, 2013 at 11:07 AM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote: >> Back when we used CVS for quite a few years I kept 7 day rolling >> snapshots of the CVS repo, against just such a difficulty as this. But >> we seem to be much better organized with infrastructure these days so I >> haven't done that for a long time. > > well there is always room for improvement(and for learning from others) > - but I agree that this proposal seems way overkill. If people think we > should keep online "delayed" mirrors we certainly have the resources to > do that on our own if we want though... What about rdiff-backup? I've set it up for personal use years ago (via the handy open source bash script backupninja) years ago and it has a pretty nice no-frills point-in-time, self-expiring, file-based automatic backup program that works well with file synchronization like rsync (I rdiff-backup to one disk and rsync the entire rsync-backup output to another disk). I've enjoyed using it quite a bit during my own personal-computer emergencies and thought the maintenance required from me has been zero, and I have used it from time to time to restore, proving it even works. Hardlinks can be used to tag versions of a file-directory tree recursively relatively compactly. It won't be as compact as a git-aware solution (since git tends to to rewrite entire files, which will confuse file-based incremental differential backup), but the amount of data we are talking about is pretty small, and as far as a lowest-common-denominator tradeoff for use in emergencies, I have to give it a lot of praise. The main advantage it has here is it implements point-in-time recovery operations that easy to use and actually seem to work. That said, I've mostly done targeted recoveries rather than trying to recover entire trees. -- fdr
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Lelundi 25 mars 2013 19:35:12, Daniel Farina a écrit :<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> On Mon, Mar 25, 2013 at 11:07AM, Stefan Kaltenbrunner<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> <stefan@kaltenbrunner.cc>wrote:<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> >> Back when we used CVS for quite a few years I kept 7day rolling<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;-qt-user-state:0;">> >> snapshots of the CVS repo, against just such a difficulty as this. But<pstyle=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>>> we seem to be much better organized with infrastructure these days so I<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>>> haven't done that for a long time.<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > <p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> well there is always room for improvement(and for learning from others)<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> - but I agree that this proposal seems way overkill. If people think we<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> should keep online "delayed" mirrors we certainly have the resources to<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> do that on our own if we want though...<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> <p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>What about rdiff-backup? I've set it up for personal use years ago<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> (via thehandy open source bash script backupninja) years ago and it<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> has a pretty nice no-frillspoint-in-time, self-expiring, file-based<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> automatic backup program that works well withfile synchronization<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> like rsync (I rdiff-backup to one disk and rsync the entire<pstyle=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;-qt-user-state:0;">> rsync-backup output to another disk). I've enjoyed using it quite a<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>bit during my own personal-computer emergencies and thought the<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> maintenancerequired from me has been zero, and I have used it from<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> time to time to restore, provingit even works. Hardlinks can be used<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> to tag versions of a file-directory tree recursivelyrelatively<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> compactly.<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> <p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>It won't be as compact as a git-aware solution (since git tends to to<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> rewriteentire files, which will confuse file-based incremental<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> differential backup), butthe amount of data we are talking about is<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> pretty small, and as far as a lowest-common-denominatortradeoff for<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> use in emergencies, I have to give it a lot of praise. The main<pstyle=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;-qt-user-state:0;">> advantage it has here is it implements point-in-time recovery<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>operations that easy to use and actually seem to work. That said,<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> I've mostlydone targeted recoveries rather than trying to recover<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> entire trees.<p style="-qt-paragraph-type:empty;margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; "> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">I have the same set up, and same feedback.<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">--<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">Cédric Villemain +33 (0)6 20 30 22 52<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">http://2ndQuadrant.fr/<pstyle=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">PostgreSQL: Support 24x7 - Développement, Expertiseet Formation
On 3/26/13 6:42 AM, Cédric Villemain wrote: > Le lundi 25 mars 2013 19:35:12, Daniel Farina a écrit : > > > On Mon, Mar 25, 2013 at 11:07 AM, Stefan Kaltenbrunner > > > > > > <stefan@kaltenbrunner.cc> wrote: > > > >> Back when we used CVS for quite a few years I kept 7 day rolling > > > >> snapshots of the CVS repo, against just such a difficulty as this. But > > > >> we seem to be much better organized with infrastructure these days so I > > > >> haven't done that for a long time. > > > > > > > > well there is always room for improvement(and for learning from others) > > > > - but I agree that this proposal seems way overkill. If people think we > > > > should keep online "delayed" mirrors we certainly have the resources to > > > > do that on our own if we want though... > > > > > > What about rdiff-backup? I've set it up for personal use years ago > > > (via the handy open source bash script backupninja) years ago and it > > > has a pretty nice no-frills point-in-time, self-expiring, file-based > > > automatic backup program that works well with file synchronization > > > like rsync (I rdiff-backup to one disk and rsync the entire > > > rsync-backup output to another disk). I've enjoyed using it quite a > > > bit during my own personal-computer emergencies and thought the > > > maintenance required from me has been zero, and I have used it from > > > time to time to restore, proving it even works. Hardlinks can be used > > > to tag versions of a file-directory tree recursively relatively > > > compactly. > > > > > > It won't be as compact as a git-aware solution (since git tends to to > > > rewrite entire files, which will confuse file-based incremental > > > differential backup), but the amount of data we are talking about is > > > pretty small, and as far as a lowest-common-denominator tradeoff for > > > use in emergencies, I have to give it a lot of praise. The main > > > advantage it has here is it implements point-in-time recovery > > > operations that easy to use and actually seem to work. That said, > > > I've mostly done targeted recoveries rather than trying to recover > > > entire trees. > > I have the same set up, and same feedback. I had the same setup, but got tired of how rdiff-backup behaved when a backup was interrupted (very lengthy cleanup process).Since then I've switched to an rsync setup that does essentially the same thing as rdiff-backup (uses hardlinksbetween multiple copies). The only downside I'm aware of is that my rsync backups aren't guaranteed to be "consistent" (for however consistent a backupof an active FS would be...). -- Jim C. Nasby, Data Architect jim@nasby.net 512.569.9461 (cell) http://jim.nasby.net
On Wed, Apr 3, 2013 at 6:18 PM, Jim Nasby <jim@nasby.net> wrote: >> > What about rdiff-backup? I've set it up for personal use years ago >> >> > (via the handy open source bash script backupninja) years ago and it >> >> > has a pretty nice no-frills point-in-time, self-expiring, file-based >> >> > automatic backup program that works well with file synchronization >> >> > like rsync (I rdiff-backup to one disk and rsync the entire >> >> > rsync-backup output to another disk). I've enjoyed using it quite a >> >> > bit during my own personal-computer emergencies and thought the >> >> > maintenance required from me has been zero, and I have used it from >> >> > time to time to restore, proving it even works. Hardlinks can be used >> >> > to tag versions of a file-directory tree recursively relatively >> >> > compactly. >> >> > >> >> > It won't be as compact as a git-aware solution (since git tends to to >> >> > rewrite entire files, which will confuse file-based incremental >> >> > differential backup), but the amount of data we are talking about is >> >> > pretty small, and as far as a lowest-common-denominator tradeoff for >> >> > use in emergencies, I have to give it a lot of praise. The main >> >> > advantage it has here is it implements point-in-time recovery >> >> > operations that easy to use and actually seem to work. That said, >> >> > I've mostly done targeted recoveries rather than trying to recover >> >> > entire trees. >> >> I have the same set up, and same feedback. > > > I had the same setup, but got tired of how rdiff-backup behaved when a > backup was interrupted (very lengthy cleanup process). Since then I've > switched to an rsync setup that does essentially the same thing as > rdiff-backup (uses hardlinks between multiple copies). > > The only downside I'm aware of is that my rsync backups aren't guaranteed to > be "consistent" (for however consistent a backup of an active FS would > be...). I forgot to add one more thing to my first mail, although it's very important to my feeble recommendation: the problem is that blind synchronization is a great way to propagate destruction. rdiff-backup (but perhaps others, too) has a file/directory structure that is, as far as I know, additive, and the pruning can be done independently at different replicas that can have different retention...and if done just right (I'm not sure about the case of concurrent backups being taken) one can write a re-check that no files are to be modified or deleted by the synchronization as a safeguard. -- fdr