Re: WIP/PoC for parallel backup - Mailing list pgsql-hackers

From Asif Rehman
Subject Re: WIP/PoC for parallel backup
Date
Msg-id CADM=JejKXdfFozc4e26V06R+y_X6c_CLE6bG0Qt+TfL2hCVkTw@mail.gmail.com
Whole thread Raw
In response to Re: WIP/PoC for parallel backup  (Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>)
List pgsql-hackers


On Tue, 21 Apr 2020 at 2:36 PM, Jeevan Ladhe <jeevan.ladhe@enterprisedb.com> wrote:
Hi Asif,

On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman <asifr.rehman@gmail.com> wrote:
Hi,

I did some tests a while back, and here are the results. The tests were done to simulate
a live database environment using pgbench.

machine configuration used for this test:
Instance Type:    t2.xlarge
Volume Type  :    io1
Memory (MiB) :    16384
vCPU #           :    4
Architecture    :    X86_64
IOP                 :    16000
Database Size (GB) :    102

The setup consist of 3 machines.
- one for database instances
- one for pg_basebackup client and
- one for pgbench with some parallel workers, simulating SELECT loads.

                                   basebackup | 4 workers | 8 Workers  | 16 workers
Backup Duration(Min):       69.25    |  20.44      | 19.86          | 20.15 
(pgbench running with 50 parallel client simulating SELECT load)


Well that looks a bit strange. All 4, 8 and 16 workers backup configurations
seem to have taken the same time. Is it because the machine CPUs are
only 4? In that case did you try to run with 2-workers and compare that
with 4-workers time?

Also, just to clarify and be sure - was there anything else running on any of
these 3 machines while the backup was in progress.

The tests were performed only for 4, 8 and 16 at the time and there was nothing else running on any of the machines.


Regards,
Jeevan Ladhe


Backup Duration(Min):       154.75   |  49.28     | 45.27         | 20.35
(pgbench running with 100 parallel client simulating SELECT load)



On Tue, Apr 21, 2020 at 9:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Apr 14, 2020 at 8:07 PM Asif Rehman <asifr.rehman@gmail.com> wrote:

I forgot to make a check for no-manifest. Fixed. Attached is the updated patch.


Have we done any performance testing with this patch to see the benefits? If so, can you point me to the results? If not, then can we perform some tests on large backups to see the benefits of this patch/idea?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


--
--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca

--
--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca

pgsql-hackers by date:

Previous
From: Thomas Munro
Date:
Subject: Re: fixing old_snapshot_threshold's time->xid mapping
Next
From: Alexander Korotkov
Date:
Subject: Re: Fix for pg_statio_all_tables