Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times - Mailing list pgsql-admin

From Scott Whitney
Subject Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times
Date
Msg-id 1652559419.2123167.1430494173076.JavaMail.zimbra@mail.int.journyx.com
Whole thread Raw
In response to Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times  (Scott Ribe <scott_ribe@elevated-dev.com>)
List pgsql-admin
A few years back, I wrote something very similar to what ended up being in PG, and I've been using it successfully ever since.

I did find there's a law of diminishing returns after about 6 threads running pg_dump.

In my case, I'm on a 6-drive SSD RAID and getting nowhere near the IOPS of the RAID,
but my suspicion is either that I'm maxing my RAID controller's potential or pg_dump just
won't get any faster.

16 cores 3.4GHz in my case.

Now, the interesting thing is that I tried this on spindle arrays, single drives, multiple machines (quad core,
quad proc, dual core, dual core-dual proc, etc) and there always seemed to be a diminishing return around
6 or 8 jobs running.

----- On May 1, 2015, at 10:22 AM, Scott Ribe <scott_ribe@elevated-dev.com> wrote:
On May 1, 2015, at 9:16 AM, Susan K. McClure <smcclure@rice.edu> wrote:
>
> The DB in question is ~25GB.  The processor has 24 Cpus, 12 cores
>  
> I have tried with "--jobs = 8, 12, and 20"  with little or no discernible improvements.

I would immediately suspect that you might be disk-bound instead of CPU-bound. What’s the setup of your storage system?

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
https://www.linkedin.com/in/scottribe/
(303) 722-0567 voice







--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin



Journyx, Inc.
7600 Burnet Road #300
Austin, TX 78757
www.journyx.com

p 512.834.8888 
f 512-834-8858 

Do you receive our promotional emails? You can subscribe or unsubscribe to those emails here


pgsql-admin by date:

Previous
From: Scott Ribe
Date:
Subject: Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times
Next
From: Scott Marlowe
Date:
Subject: Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times