Postgres Pro
Downloads
Home   >   mailing lists

Thread: files to ignore during pitr base backup

files to ignore during pitr base backup

From
Jeff Frost
Date:
I noticed recently that my PITR base backups were occassionally returning
non-zero, so I turned up the verbosity on rsync and it appears that
occassionally a file vanishes out from under rsync while it's doing the base
backup:

          700   4%   52.58kB/s    0:00:00
        16384 100%    1.12MB/s    0:00:00  (125, 71.4% of 1024)
file has vanished: "/usr/local/pgsql/data/base/1219609879/pg_internal.init"
base/17229/

sent 258563798 bytes  received 1348515 bytes  2809862.84 bytes/sec
total size is 8273133704  speedup is 31.83
rsync warning: some files vanished before they could be transferred (code
24) at main.c(791)
PITR base backup rsync command failed!

Would it be ok to just exclude the pg_internal.init file since it disappeared
during the base backup?

--
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954

Re: files to ignore during pitr base backup

From
Tom Lane
Date:
Jeff Frost <jeff@frostconsultingllc.com> writes:
> I noticed recently that my PITR base backups were occassionally returning
> non-zero, so I turned up the verbosity on rsync and it appears that
> occassionally a file vanishes out from under rsync while it's doing the base
> backup:
> ...
> Would it be ok to just exclude the pg_internal.init file since it disappeared
> during the base backup?

It's safe to exclude those, but those are hardly the only files that
might "disappear" in a live database.  It'd be better to use a dump
tool that's less picky about the source data changing under it.  We've
had similar reports about GNU tar being too smart for this purpose :-(

            regards, tom lane

Re: files to ignore during pitr base backup

From
Jeff Frost
Date:
On Tue, 25 Oct 2005, Tom Lane wrote:

> It's safe to exclude those, but those are hardly the only files that
> might "disappear" in a live database.  It'd be better to use a dump
> tool that's less picky about the source data changing under it.  We've
> had similar reports about GNU tar being too smart for this purpose :-(

I would have thought that the pg_start_backup() function would tell the
postmasters not to take files out from under the base backup process?

In any event, if it's ok to ignore those, rsync conveniently has a specific
error code for that:

        24     Partial transfer due to vanished source files

So I can just consider an exit code of 0 or 24 as both being success.

--
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954

Re: files to ignore during pitr base backup

From
Alvaro Herrera
Date:
Jeff Frost wrote:
> On Tue, 25 Oct 2005, Tom Lane wrote:
>
> >It's safe to exclude those, but those are hardly the only files that
> >might "disappear" in a live database.  It'd be better to use a dump
> >tool that's less picky about the source data changing under it.  We've
> >had similar reports about GNU tar being too smart for this purpose :-(
>
> I would have thought that the pg_start_backup() function would tell the
> postmasters not to take files out from under the base backup process?

That would be impossible unless you wanted backends to freeze from time
to time while you take the base backup.

--
Alvaro Herrera                  http://www.amazon.com/gp/registry/5ZYLFMCVHXC
"La rebeldía es la virtud original del hombre" (Arthur Schopenhauer)