Re: pg_restore --multi-thread - Mailing list pgsql-hackers

From Andrew Dunstan
Subject Re: pg_restore --multi-thread
Date
Msg-id 499EE997.40409@dunslane.net
Whole thread Raw
In response to Re: pg_restore --multi-thread  ("Joshua D. Drake" <jd@commandprompt.com>)
Responses Re: pg_restore --multi-thread  ("Kevin Grittner" <Kevin.Grittner@wicourts.gov>)
List pgsql-hackers

Joshua D. Drake wrote:
> On Fri, 2009-02-20 at 09:33 -0500, Andrew Dunstan wrote:
>
>   
>> The short answer is that we don't know yet. There is anecdotal evidence 
>> that the number of CPUs on the server is a good place to start, but we 
>> should be honest enough to say that this is a new feature and we are 
>> still gathering information about its performance.  If you want to give 
>> some advice, then I think the best advice is to try a variety of 
>> settings to see what works best for you, and if you have a good set of 
>> figures report it back to us.
>>     
>
> There has been some fairly heavy testing and research that caused the
> patch in the first place. The thread is here:
>
> http://archives.postgresql.org/pgsql-hackers/2008-02/msg00695.php
>
> It is a long thread. The end was result was the fastest restore time for
> 220G was performed with 24 threads with an 8 core box. It came in at 3.5
> hours.
>
> http://archives.postgresql.org/pgsql-hackers/2008-02/msg01092.php
>
> It is important to point out that this was a machine with 50 spindles.
> Which is where your bottleneck is going to be immediately after solving
> the CPU bound nature of the problem.
>
> So although the CPU question is easily answered, the IO is not. IO is
> extremely variable in its performance.
>
>
>   

Yes, quite true. But parallel restore doesn't work quite the same way 
your original shell scripts did. It tries harder to keep the job pool 
continuously occupied, and so its best number of jobs is likely to be a 
bit lower then yours.

But you are right that there isn't a simple formula.

cheers

andrew


pgsql-hackers by date:

Previous
From: "Joshua D. Drake"
Date:
Subject: Re: pg_restore --multi-thread
Next
From: Kenneth Marshall
Date:
Subject: Re: pg_restore --multi-thread