Re: parallelizing the archiver - Mailing list pgsql-hackers

From Stephen Frost
Subject Re: parallelizing the archiver
Date
Msg-id 20210914201458.GH17906@tamriel.snowman.net
Whole thread Raw
In response to Re: parallelizing the archiver  (Julien Rouhaud <rjuju123@gmail.com>)
Responses Re: parallelizing the archiver  (Julien Rouhaud <rjuju123@gmail.com>)
List pgsql-hackers
Greetings,

* Julien Rouhaud (rjuju123@gmail.com) wrote:
> On Fri, Sep 10, 2021 at 2:03 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> > > 10 сент. 2021 г., в 10:52, Julien Rouhaud <rjuju123@gmail.com> написал(а):
> > > Yes, but it also means that it's up to every single archiving tool to
> > > implement a somewhat hackish parallel version of an archive_command,
> > > hoping that core won't break it.

We've got too many archiving tools as it is, if you want my 2c on that.

> > I'm not proposing to remove existing archive_command. Just deprecate it one-WAL-per-call form.
>
> Which is a big API beak.

We definitely need to stop being afraid of this.  We completely changed
around how restores work and made pretty much all of the backup/restore
tools have to make serious changes when we released v12.

I definitely don't think that we should be making assumptions that
changing archive command to start running things in parallel isn't
*also* an API break too, in any case.  It is also a change and there's
definitely a good chance that it'd break some of the archivers out
there.  If we're going to make a change here, let's make a sensible one.

> > It's a very simplistic approach. If some GUC is set - archiver will just feed ready files to stdin of archive
command.What fundamental design changes we need? 

Haven't really thought about this proposal but it does sound
interesting.

Thanks,

Stephen

Attachment

pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Getting ERROR "subplan "SubPlan 1" was not initialized" in EXISTS subplan when using for list partition.
Next
From: Jelte Fennema
Date:
Subject: Re: [EXTERNAL] Re: Don't clean up LLVM state when exiting in a bad way