Re: parallelizing the archiver - Mailing list pgsql-hackers
| From | Magnus Hagander |
|---|---|
| Subject | Re: parallelizing the archiver |
| Date | |
| Msg-id | CABUevEx8cKy=+YQU_3NaeXnZV2bSB7Lk6EE+-FEcmE4JO4V1hg@mail.gmail.com Whole thread Raw |
| In response to | Re: parallelizing the archiver (Robert Haas <robertmhaas@gmail.com>) |
| Responses |
Re: parallelizing the archiver
("Bossart, Nathan" <bossartn@amazon.com>)
Re: parallelizing the archiver (Robert Haas <robertmhaas@gmail.com>) Re: parallelizing the archiver (Stephen Frost <sfrost@snowman.net>) |
| List | pgsql-hackers |
On Tue, Oct 19, 2021 at 2:50 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Oct 18, 2021 at 7:25 PM Bossart, Nathan <bossartn@amazon.com> wrote:
> I think the biggest question is where to put the archive_command
> module, which I've called shell_archive. The only existing directory
> that looked to me like it might work is src/test/modules. It might be
> rather bold to relegate this functionality to a test module so
> quickly, but on the other hand, perhaps it's the right thing to do
> given we intend to deprecate it in the future. I'm curious what
> others think about this.
I don't see that as being a viable path forward based on my customer
interactions working here at EDB.
I am not quite sure why we wouldn't just compile the functions into
the server. Functions pointers can point to core functions as surely
as loadable modules. The present design isn't too congenial to that
because it's relying on the shared library loading mechanism to wire
the thing in place - but there's no reason it has to be that way.
Logical decoding plugins don't work that way, for example. We could
still have a GUC, say call it archive_method, that selects the module
-- with 'shell' being a builtin method, and others being loadable as
modules. If you set archive_method='shell' then you enable this
module, and it has its own GUC, say call it archive_command, to
configure the behavior.
Yeah, seems reasonable. It wouldn't serve as well as an example to developers, but then it's probably not the "loadable module" part of building it that people need examples of. So as long as it's using the same internal APIs and just happens to be compiled in by default, I see no problem with that.
But, is logical decoding really that great an example? I mean, we build pgoutput.so as a library, we don't provide it compiled-in. So we could build the "shell archiver" based on that pattern, in which case we should create a postmaster/shell_archiver directory or something like that?
It should definitely not go under "test".
An advantage of this approach is that it's perfectly
backward-compatible. I understand that archive_command is a hateful
thing to many people here, but software has to serve the user base,
not just the developers. Lots of people use archive_command and rely
on it -- and are not interested in installing yet another piece of
out-of-core software to do what $OTHERDB has built in.
Backwards compatibility is definitely a must, I'd say. Regardless of exactly how the backwards-compatible pugin is shipped, it should be what's turned on by default.
pgsql-hackers by date: