Re: parallelizing the archiver - Mailing list pgsql-hackers

From Stephen Frost
Subject Re: parallelizing the archiver
Date
Msg-id 20211005031812.GP20998@tamriel.snowman.net
Whole thread Raw
In response to Re: parallelizing the archiver  ("Bossart, Nathan" <bossartn@amazon.com>)
Responses Re: parallelizing the archiver  ("Bossart, Nathan" <bossartn@amazon.com>)
List pgsql-hackers
Greetings,

* Bossart, Nathan (bossartn@amazon.com) wrote:
> On 10/4/21, 7:21 PM, "Stephen Frost" <sfrost@snowman.net> wrote:
> > This has something we've contemplated quite a bit and the last thing
> > that I'd want to have is a requirement to configure a whole bunch of
> > additional parameters to enable this.  Why do we need to have some many
> > new GUCs?  I would have thought we'd probably be able to get away with
> > just having the appropriate hooks and then telling folks to load the
> > extension in shared_preload_libraries..
>
> That would certainly simplify my patch quite a bit.  I'll do it this
> way in the next revision.
>
> > As for the hooks themselves, I'd certainly hope that they'd be designed
> > to handle batches of WAL rather than individual ones as that's long been
> > one of the main issues with the existing archive command approach.  I
> > appreciate that maybe that's less of an issue with a shared library but
> > it's still something to consider.
>
> Will do.  This seems like it should be easier with the hook because we
> can provide a way to return which files were successfully archived.

It's also been discussed, at least around the water cooler (as it were
in pandemic times- aka our internal slack channels..) that the existing
archive command might be reimplemented as an extension using these.  Not
sure if that's really necessary but it was a thought.  In any case,
thanks for working on this!

Stephen

Attachment

pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: plperl: update ppport.h and fix configure version check
Next
From: Mark Dilger
Date:
Subject: Re: BUG #17212: pg_amcheck fails on checking temporary relations