Re: Further pg_upgrade analysis for many tables - Mailing list pgsql-hackers

From Bruce Momjian
Subject Re: Further pg_upgrade analysis for many tables
Date
Msg-id 20130123032457.GA22758@momjian.us
Whole thread Raw
In response to Re: Further pg_upgrade analysis for many tables  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
On Sun, Jan 20, 2013 at 02:11:48PM -0500, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > !          *    Using pg_restore --single-transaction is faster than other
> > !          *    methods, like --jobs.
>
> Is this still the case now that Jeff's AtEOXact patch is in?  The risk
> of locktable overflow with --single-transaction makes me think that
> pg_upgrade should avoid it unless there is a *really* strong performance
> case for it, and I fear your old measurements are now invalidated.

I had thought that the AtEOXact patch only helped single transactions
with many tables, but I now remember it mostly helps backends that have
accessed many tables.

With max_locks_per_transaction set high, I tested with the attached
patch that removes --single-transaction from pg_restore.  I saw a 4%
improvement by removing that option, and 15% at 64k.  (Test script
attached.)  I have applied the patch.  This is good news not just for
pg_upgrade but for other backends that access many tables.

                 git     patch
        1       11.06    11.03
     1000       19.97    20.86
     2000       28.50    27.61
     4000       46.90    45.65
     8000       79.38    80.68
    16000      153.33   147.13
    32000      317.40   302.96
    64000      782.94   659.52

FYI, this is better than the tests I did on the original patch that
showed --single-transaction was still a win then:

    http://www.postgresql.org/message-id/20121128202232.GA31741@momjian.us

>     #tbls       git     -1    AtOEXAct  both
>         1      11.06   13.06   10.99   13.20
>      1000      21.71   22.92   22.20   22.51
>      2000      32.86   31.09   32.51   31.62
>      4000      55.22   49.96   52.50   49.99
>      8000     105.34   82.10   95.32   82.94
>     16000     223.67  164.27  187.40  159.53
>     32000     543.93  324.63  366.44  317.93
>     64000    1697.14  791.82  767.32  752.57

Keep in mind this doesn't totally avoid the requirement to increase
max_locks_per_transaction.  There are cases at >6k where pg_dump runs
out of locks, but I don't see how we can improve that.  Hopefully users
have already seen pg_dump fail and have adjusted
max_locks_per_transaction.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Attachment

pgsql-hackers by date:

Previous
From: Gavin Flower
Date:
Subject: Re: Event Triggers: adding information
Next
From: Amit Kapila
Date:
Subject: Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]