Re: Would it be possible to have parallel archiving? - Mailing list pgsql-hackers

From Stephen Frost
Subject Re: Would it be possible to have parallel archiving?
Date
Msg-id 20180828212010.GN3326@tamriel.snowman.net
Whole thread Raw
In response to Re: Would it be possible to have parallel archiving?  (David Steele <david@pgmasters.net>)
Responses Re: Would it be possible to have parallel archiving?
List pgsql-hackers
Greetings,

* David Steele (david@pgmasters.net) wrote:
> On 8/28/18 4:34 PM, Andrey Borodin wrote:
> >>
> >> I still don't think it's a good idea and I specifically recommend
> >> against making changes to the archive status files- those are clearly
> >> owned and managed by PG and should not be whacked around by external
> >> processes.
> > If you do not write to archive_status, you basically have two options:
> > 1. On every archive_command recheck that archived file is identical to file that is already archived. This hurts
performance.
> > 2. Hope that files match. This does not add any safety compared to whacking archive_status. This approach is prone
tocore  changes as writes are. 
>
> Another option is to maintain the state of what has been safely archived
> (and what has errored) locally.  This allows pgBackRest to rapidly
> return the status to Postgres without rechecking against the repository,
> which as you note would be very slow.
>
> This allows more than one archive_command to be safely run since all
> archive commands must succeed before Postgres will mark the segment as done.
>
> It's true that reading archive_status is susceptible to core changes but
> the less interaction the better, I think.

Absolutely.  External processes shouldn't be changing the files written
out and managed by the core system.  pgbackrest is *much* safer than
alternatives which hack around files inside of PGDATA.  We've been
working to move things forward to the point where pgbackrest is able to
be run as a user who hasn't even got access to modify those files (which
has now landed in PG11) and for good reason- it's outright dangerous to
do.

Thanks!

Stephen

Attachment

pgsql-hackers by date:

Previous
From: David Steele
Date:
Subject: Re: Would it be possible to have parallel archiving?
Next
From: Tom Lane
Date:
Subject: Re: some pg_dump query code simplification