Thread: PG11 Parallel Thanks!!

PG11 Parallel Thanks!!

From
Jason Ralph
Date:
I wanted to drop a quick note thanking the developers who have contributed to Postgres.  I have recently upgraded our
productionPG instances from pg9.3 to pg11. 

We do a lot of table syncs, and we have one process at the end of the month that syncs 3 very large tables (400GB).
Thissync happens from a shell script using pg_dump and pg_restore, we have it set to use -j3 but it's called
sequentiallyso it never really takes advantage of parallel. 

Since pg11 on both the target and source, the run time has decreased a lot, I chalk it up to the parallel index
creationsin pg11 which was a very time consuming process on pg9.3. 
The process has finished almost 10 hours earlier than pg93.  So thank you for your hard work and dedication to this
awesomepiece of software. 



Jason Ralph


This message contains confidential information and is intended only for the individual named. If you are not the named
addresseeyou should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if
youhave received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be
guaranteedto be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or
incomplete,or contain viruses. The sender therefore does not accept liability for any errors or omissions in the
contentsof this message, which arise as a result of e-mail transmission. If verification is required please request a
hard-copyversion. 



Re: PG11 Parallel Thanks!!

From
Pankaj Jangid
Date:
Jason Ralph <jralph@affinitysolutions.com> writes:

> I wanted to drop a quick note thanking the developers who have
> contributed to Postgres.  I have recently upgraded our production PG
> instances from pg9.3 to pg11.

>
> We do a lot of table syncs, and we have one process at the end of the
> month that syncs 3 very large tables (400GB).  This sync happens from
> a shell script using pg_dump and pg_restore, we have it set to use -j3
> but it's called sequentially so it never really takes advantage of
> parallel.
>
> Since pg11 on both the target and source, the run time has decreased a
> lot, I chalk it up to the parallel index creations in pg11 which was a
> very time consuming process on pg9.3.
> The process has finished almost 10 hours earlier than pg93.  So thank
> you for your hard work and dedication to this awesome piece of
> software.

This is a great user story. Thanks for sharing your experience, Jason.

-- 
Pankaj Jangid



Re: PG11 Parallel Thanks!!

From
Peter Geoghegan
Date:
On Wed, Oct 2, 2019 at 8:41 AM Jason Ralph <jralph@affinitysolutions.com> wrote:
> Since pg11 on both the target and source, the run time has decreased a lot, I chalk it up to the parallel index
creationsin pg11 which was a very time consuming process on pg9.3.
 
> The process has finished almost 10 hours earlier than pg93.  So thank you for your hard work and dedication to this
awesomepiece of software.
 

How long did it take on 9.3?

I am the author of the parallel CREATE INDEX feature. It's good to get
feedback like this.

-- 
Peter Geoghegan



RE: PG11 Parallel Thanks!!

From
Jason Ralph
Date:
>On Wed, Oct 2, 2019 at 8:41 AM Jason Ralph <jralph@affinitysolutions.com> wrote:
> Since pg11 on both the target and source, the run time has decreased a lot, I chalk it up to the parallel index
creationsin pg11 which was a very time consuming >process on pg9.3.
 
> The process has finished almost 10 hours earlier than pg93.  So thank you for your hard work and dedication to this
awesomepiece of software.
 

>How long did it take on 9.3?

>I am the author of the parallel CREATE INDEX feature. It's good to get feedback like this.

>--
>Peter Geoghegan


Hello @Peter Geoghegan,
Thank you!
The end of month process that we run at my company was a pg_dump and pg_restore of 3 tables, these tables are around
~(400GB)each.  The entire process on pg93 took 29 hours.
 

The index creation portion of the restore on the target pg9.3 database took:
5) time: -15 hours -4 minute ((-54264 % 60)) seconds

The index creation of the restore on the target db after pg11 upgrade on source and target took:
 5) time: -5 hours -7 minute ((-18434 % 60)) seconds

We saved 10 hours!!
Here is what we have on the target pg11 database., so max_parallel_workers_per_gather = 3 is what I think was what
reducedthe index creation time.
 

[13:20:04] [root@database ~] # cat /proc/cpuinfo | grep processor
processor: 0
processor: 1
processor: 2
processor: 3
processor: 4
processor: 5

max_worker_processes = 6                # (change requires restart)
max_parallel_workers_per_gather = 3     # taken from max_parallel_workers
max_parallel_workers = 6

Jason Ralph




-----Original Message-----
From: Peter Geoghegan <pg@bowt.ie>
Sent: Thursday, October 3, 2019 1:26 AM
To: Jason Ralph <jralph@affinitysolutions.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: PG11 Parallel Thanks!!

On Wed, Oct 2, 2019 at 8:41 AM Jason Ralph <jralph@affinitysolutions.com> wrote:
> Since pg11 on both the target and source, the run time has decreased a lot, I chalk it up to the parallel index
creationsin pg11 which was a very time consuming process on pg9.3.
 
> The process has finished almost 10 hours earlier than pg93.  So thank you for your hard work and dedication to this
awesomepiece of software.
 

How long did it take on 9.3?

I am the author of the parallel CREATE INDEX feature. It's good to get feedback like this.

--
Peter Geoghegan
This message contains confidential information and is intended only for the individual named. If you are not the named
addresseeyou should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if
youhave received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be
guaranteedto be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or
incomplete,or contain viruses. The sender therefore does not accept liability for any errors or omissions in the
contentsof this message, which arise as a result of e-mail transmission. If verification is required please request a
hard-copyversion.
 

Re: PG11 Parallel Thanks!!

From
Peter Geoghegan
Date:
On Thu, Oct 3, 2019 at 10:31 AM Jason Ralph
<jralph@affinitysolutions.com> wrote:
> The end of month process that we run at my company was a pg_dump and pg_restore of 3 tables, these tables are around
~(400GB)each.  The entire process on pg93 took 29 hours.
 
>
> The index creation portion of the restore on the target pg9.3 database took:
> 5) time: -15 hours -4 minute ((-54264 % 60)) seconds
>
> The index creation of the restore on the target db after pg11 upgrade on source and target took:
>  5) time: -5 hours -7 minute ((-18434 % 60)) seconds
>
> We saved 10 hours!!

The sort code received many improvements over the years, really
starting in 9.5, and continuing in 9.6, 10 and 11. FWIW, I think that
that was probably the biggest factor here. Though parallel CREATE
INDEX will have helped as well.

-- 
Peter Geoghegan