Re: parallelizing the archiver - Mailing list pgsql-hackers

From Robert Haas
Subject Re: parallelizing the archiver
Date
Msg-id CA+Tgmobhyc52NDZxJTjkX2CXftrZGfyuuuNS=6zxihW+ObfeTA@mail.gmail.com
Whole thread Raw
In response to Re: parallelizing the archiver  ("Bossart, Nathan" <bossartn@amazon.com>)
Responses Re: parallelizing the archiver  (David Steele <david@pgmasters.net>)
Re: parallelizing the archiver  (Magnus Hagander <magnus@hagander.net>)
List pgsql-hackers
On Mon, Oct 18, 2021 at 7:25 PM Bossart, Nathan <bossartn@amazon.com> wrote:
> I think the biggest question is where to put the archive_command
> module, which I've called shell_archive.  The only existing directory
> that looked to me like it might work is src/test/modules.  It might be
> rather bold to relegate this functionality to a test module so
> quickly, but on the other hand, perhaps it's the right thing to do
> given we intend to deprecate it in the future.  I'm curious what
> others think about this.

I don't see that as being a viable path forward based on my customer
interactions working here at EDB.

I am not quite sure why we wouldn't just compile the functions into
the server. Functions pointers can point to core functions as surely
as loadable modules. The present design isn't too congenial to that
because it's relying on the shared library loading mechanism to wire
the thing in place - but there's no reason it has to be that way.
Logical decoding plugins don't work that way, for example. We could
still have a GUC, say call it archive_method, that selects the module
-- with 'shell' being a builtin method, and others being loadable as
modules. If you set archive_method='shell' then you enable this
module, and it has its own GUC, say call it archive_command, to
configure the behavior.

An advantage of this approach is that it's perfectly
backward-compatible. I understand that archive_command is a hateful
thing to many people here, but software has to serve the user base,
not just the developers. Lots of people use archive_command and rely
on it -- and are not interested in installing yet another piece of
out-of-core software to do what $OTHERDB has built in.

-- 
Robert Haas
EDB: http://www.enterprisedb.com



pgsql-hackers by date:

Previous
From: Amit Kapila
Date:
Subject: Re: Added schema level support for publication.
Next
From: Robert Haas
Date:
Subject: Re: .ready and .done files considered harmful