Re: Would it be possible to have parallel archiving? - Mailing list pgsql-hackers

From Stephen Frost
Subject Re: Would it be possible to have parallel archiving?
Date
Msg-id 20180828200754.GI3326@tamriel.snowman.net
Whole thread Raw
In response to Re: Would it be possible to have parallel archiving?  (Andrey Borodin <x4mmm@yandex-team.ru>)
Responses Re: Would it be possible to have parallel archiving?
List pgsql-hackers
Greetings,

* Andrey Borodin (x4mmm@yandex-team.ru) wrote:
> > 28 авг. 2018 г., в 14:08, Stephen Frost <sfrost@snowman.net> написал(а):
> > * David Steele (david@pgmasters.net <mailto:david@pgmasters.net>) wrote:
> >> On 8/28/18 8:32 AM, Stephen Frost wrote:
> >> To be clear, pgBackRest uses the .ready files in archive_status to
> >> parallelize archiving but still notifies PostgreSQL of completion via
> >> the archive_command mechanism.  We do not modify .ready files to .done
> >> directly.
> >
> > Right, we don't recommend mucking around with that directory of files.
> > Even if that works today (which you'd need to test extensively...),
> > there's no guarantee that it'll work and do what you want in the
> > future...
> WAL-G modifies archive_status files.

Frankly, I've heard far too many concerns and issues with WAL-G to
consider anything it does at all sensible.

> This path was chosen to limit state preserved between WAL-G runs (archiving to S3) and further push archiving
performance.

I still don't think it's a good idea and I specifically recommend
against making changes to the archive status files- those are clearly
owned and managed by PG and should not be whacked around by external
processes.

> Indeed, it was very hard to test. Also, this makes impossible to use two archiving system simultaneously for transit
period.

The testing in WAL-G seems to be rather lacking from what I've seen.

It's not clear to me what you're suggesting wrt 'two archiving system
simultaneously for transit period', but certainly whacking the archive
status files around doesn't make it easier to have multiple archive
commands trying to simultaneously archive WAL files.

Thanks!

Stephen

Attachment

pgsql-hackers by date:

Previous
From: Andrey Borodin
Date:
Subject: Re: Would it be possible to have parallel archiving?
Next
From: Andrey Borodin
Date:
Subject: Re: Would it be possible to have parallel archiving?