Re: parallelizing the archiver - Mailing list pgsql-hackers

From Andrey Borodin
Subject Re: parallelizing the archiver
Date
Msg-id FA9D5725-17CA-4476-8B3D-919BD32AFD96@yandex-team.ru
Whole thread Raw
In response to parallelizing the archiver  ("Bossart, Nathan" <bossartn@amazon.com>)
Responses Re: parallelizing the archiver  (Julien Rouhaud <rjuju123@gmail.com>)
List pgsql-hackers

> 8 сент. 2021 г., в 03:36, Bossart, Nathan <bossartn@amazon.com> написал(а):
>
> Anyway, I'm curious what folks think about this.  I think it'd help
> simplify server administration for many users.

BTW this thread is also related [0].

My 2 cents.
It's OK if external tool is responsible for concurrency. Do we want this complexity in core? Many users do not enable
archivingat all. 
Maybe just add parallelism API for external tool?
It's much easier to control concurrency in external tool that in PostgreSQL core. Maintaining parallel worker is a
tremendouslyharder than spawning goroutine, thread, task or whatever. 
External tool needs to know when xlog segment is ready and needs to report when it's done. Postgres should just ensure
thatexternal archiever\restorer is running. 
For example external tool could read xlog names from stdin and report finished files from stdout. I can prototype such
toolswiftly :) 
E.g. postgres runs ```wal-g wal-archiver``` and pushes ready segment filenames on stdin. And no more listing of
archive_statusand hacky algorithms to predict next WAL name and completition time! 

Thoughts?

Best regards, Andrey Borodin.

[0]
https://www.postgresql.org/message-id/flat/CA%2BTgmobhAbs2yabTuTRkJTq_kkC80-%2Bjw%3DpfpypdOJ7%2BgAbQbw%40mail.gmail.com


pgsql-hackers by date:

Previous
From: Dilip Kumar
Date:
Subject: Re: Toast compression method options
Next
From: "houzj.fnst@fujitsu.com"
Date:
Subject: RE: Added schema level support for publication.