Re: Extracting cross-version-upgrade knowledge from buildfarm client - Mailing list pgsql-hackers

OK, here's a v4:

* It works with 002_pg_upgrade.pl now.  The only substantive change
I had to make for that was to define the $old_version arguments as
being always PostgreSQL::Version objects not strings, because
otherwise I got complaints like

Argument "HEAD" isn't numeric in numeric comparison (<=>) at
/home/postgres/pgsql/src/bin/pg_upgrade/../../../src/test/perl/PostgreSQL/Version.pmline 130. 

So now TestUpgradeXversion.pm is responsible for performing that
conversion, and also for not doing any conversions on HEAD (which
Andrew wanted anyway).

* I improved pg_upgrade's TESTING directions after figuring out how
to get it to work for contrib modules.

* Incorporated (most of) Andrew's stylistic improvements.

* Simplified TestUpgradeXversion.pm's use of diff, as discussed.

I think we're about ready to go, except for cutting down
AdjustUpgrade.pm to make versions to put in the back branches.

I'm slightly tempted to back-patch 002_pg_upgrade.pl so that there
is an in-tree way to verify back-branch AdjustUpgrade.pm files.
On the other hand, it's hard to believe that testing that in
HEAD won't be sufficient; I doubt the back-branch copies will
need to change much.

            regards, tom lane

diff --git a/src/bin/pg_upgrade/TESTING b/src/bin/pg_upgrade/TESTING
index 98286231d7..81a4324a76 100644
--- a/src/bin/pg_upgrade/TESTING
+++ b/src/bin/pg_upgrade/TESTING
@@ -10,31 +10,14 @@ This will run the TAP tests to run pg_upgrade, performing an upgrade
 from the version in this source tree to a new instance of the same
 version.

-Testing an upgrade from a different version requires a dump to set up
-the contents of this instance, with its set of binaries.  The following
-variables are available to control the test (see DETAILS below about
-the creation of the dump):
+Testing an upgrade from a different PG version is also possible, and
+provides a more thorough test that pg_upgrade does what it's meant for.
+This requires both a source tree and an installed tree for the old
+version, as well as a dump file to set up the instance to be upgraded.
+The following environment variables must be set to enable this testing:
 export olddump=...somewhere/dump.sql    (old version's dump)
 export oldinstall=...otherversion/    (old version's install base path)
-
-"filter_rules" is a variable that can be used to specify a file with custom
-filtering rules applied before comparing the dumps of the PostgreSQL
-instances near the end of the tests, in the shape of regular expressions
-valid for perl.  This is useful to enforce certain validation cases where
-pg_dump could create inconsistent outputs across major versions.
-For example:
-
-    # Remove all CREATE POLICY statements
-    s/^CREATE\sPOLICY.*//mgx
-    # Replace REFRESH with DROP for materialized views
-    s/^REFRESH\s(MATERIALIZED\sVIEW)/DROP $1/mgx
-
-Lines beginning with '#' and empty lines are ignored.  One rule can be
-defined per line.
-
-Finally, the tests can be done by running
-
-    make check
+See DETAILS below for more information about creation of the dump.

 You can also test the different transfer modes (--copy, --link,
 --clone) by setting the environment variable PG_TEST_PG_UPGRADE_MODE
@@ -52,22 +35,32 @@ The most effective way to test pg_upgrade, aside from testing on user
 data, is by upgrading the PostgreSQL regression database.

 This testing process first requires the creation of a valid regression
-database dump that can be then used for $olddump.  Such files contain
+database dump that can then be used for $olddump.  Such files contain
 most database features and are specific to each major version of Postgres.

 Here are the steps needed to create a dump file:

 1)  Create and populate the regression database in the old cluster.
     This database can be created by running 'make installcheck' from
-    src/test/regress using its source code tree.
+    src/test/regress in the old version's source code tree.

-2)  Use pg_dumpall to dump out the contents of the instance, including the
-    regression database, in the shape of a SQL file.  This requires the *old*
-    cluster's pg_dumpall so as the dump created is compatible with the
-    version of the cluster it is dumped into.
+    If you like, you can also populate regression databases for one or
+    more contrib modules by running 'make installcheck USE_MODULE_DB=1'
+    in their directories.  (USE_MODULE_DB is essential so that the
+    pg_upgrade test script will understand which database is which.)

-Once the dump is created, it can be repeatedly used with $olddump and
-`make check`, that automates the dump of the old database, its upgrade,
-the dump out of the new database and the comparison of the dumps between
-the old and new databases.  The contents of the dumps can also be manually
-compared.
+2)  Use pg_dumpall to dump out the contents of the instance, including the
+    regression database(s), into a SQL file.  Use the *old* version's
+    pg_dumpall so that the dump created is compatible with that version.
+
+Once the dump file is created, it can be used repeatedly.  Set $olddump
+to point to the dump file and run 'make check' or 'make installcheck'
+in the new version's src/bin/pg_upgrade directory.  (If you included any
+contrib databases in the old dump, you must use 'make installcheck' and
+ensure that the corresponding contrib modules have been installed in
+the new version's installation tree.)  This will build a temporary cluster
+using the old installation's executables, populate it from the dump file,
+and then try to pg_upgrade it to the new version.  Success is reported
+if pg_dumpall output matches between the pre-upgrade and post-upgrade
+databases.  In case of trouble, manually comparing those dump files may
+help to isolate the problem.
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index c066fd7d93..62a8fa9d8b 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -10,6 +10,7 @@ use File::Path qw(rmtree);

 use PostgreSQL::Test::Cluster;
 use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::AdjustUpgrade;
 use Test::More;

 # Can be changed to test the other modes.
@@ -37,37 +38,16 @@ sub generate_db
 # This returns the path to the filtered dump.
 sub filter_dump
 {
-    my ($node, $dump_file) = @_;
+    my ($is_old, $old_version, $dump_file) = @_;
     my $dump_contents = slurp_file($dump_file);

-    # Remove the comments.
-    $dump_contents =~ s/^\-\-.*//mgx;
-    # Remove empty lines.
-    $dump_contents =~ s/^\n//mgx;
-
-    # Apply custom filtering rules, if any.
-    if (defined($ENV{filter_rules}))
+    if ($is_old)
     {
-        my $filter_file = $ENV{filter_rules};
-        die "no file with custom filter rules found!" unless -e $filter_file;
-
-        open my $filter_handle, '<', $filter_file
-          or die "could not open $filter_file";
-        while (<$filter_handle>)
-        {
-            my $filter_line = $_;
-
-            # Skip comments and empty lines
-            next if ($filter_line =~ /^#/);
-            next if ($filter_line =~ /^\s*$/);
-
-            # Apply lines with filters.
-            note "Applying custom rule $filter_line to $dump_file";
-            my $filter = "\$dump_contents =~ $filter_line";
-            ## no critic (ProhibitStringyEval)
-            eval $filter;
-        }
-        close $filter_handle;
+        $dump_contents = adjust_old_dumpfile($old_version, $dump_contents);
+    }
+    else
+    {
+        $dump_contents = adjust_new_dumpfile($old_version, $dump_contents);
     }

     my $dump_file_filtered = "${dump_file}_filtered";
@@ -83,7 +63,7 @@ sub filter_dump
 # that gets upgraded.  Before running the upgrade, a logical dump of the
 # old cluster is taken, and a second logical dump of the new one is taken
 # after the upgrade.  The upgrade test passes if there are no differences
-# in these two dumps.
+# (after filtering) in these two dumps.

 # Testing upgrades with an older version of PostgreSQL requires setting up
 # two environment variables, as of:
@@ -198,15 +178,29 @@ my $oldbindir = $oldnode->config_data('--bindir');
 # only if different major versions are used for the dump.
 if (defined($ENV{oldinstall}))
 {
-    # Note that upgrade_adapt.sql and psql from the new version are used,
-    # to cope with an upgrade to this version.
-    $newnode->command_ok(
-        [
-            'psql', '-X',
-            '-f',   "$srcdir/src/bin/pg_upgrade/upgrade_adapt.sql",
-            '-d',   $oldnode->connstr('regression'),
-        ],
-        'ran adapt script');
+    # Consult AdjustUpgrade to find out what we need to do.
+    my $dbnames =
+      $oldnode->safe_psql('postgres', qq(SELECT datname FROM pg_database));
+    my %dbnames;
+    do { $dbnames{$_} = 1; }
+      foreach split /\s+/s, $dbnames;
+    my $adjust_cmds =
+      adjust_database_contents($oldnode->pg_version, %dbnames);
+
+    foreach my $updb (keys %$adjust_cmds)
+    {
+        my $upcmds = join(";\n", @{ $adjust_cmds->{$updb} });
+
+        # For simplicity, use the newer version's psql to issue the commands.
+        $newnode->command_ok(
+            [
+                'psql', '-X',
+                '-v',   'ON_ERROR_STOP=1',
+                '-c',   $upcmds,
+                '-d',   $oldnode->connstr($updb),
+            ],
+            "ran version adaptation commands for database $updb");
+    }
 }

 # Take a dump before performing the upgrade as a base comparison. Note
@@ -359,8 +353,8 @@ my $dump1_filtered = $dump1_file;
 my $dump2_filtered = $dump2_file;
 if ($oldnode->pg_version != $newnode->pg_version)
 {
-    $dump1_filtered = filter_dump($oldnode, $dump1_file);
-    $dump2_filtered = filter_dump($newnode, $dump2_file);
+    $dump1_filtered = filter_dump(1, $oldnode->pg_version, $dump1_file);
+    $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
 }

 # Compare the two dumps, there should be no differences.
@@ -371,7 +365,7 @@ is($compare_res, 0, 'old and new dumps match after pg_upgrade');
 if ($compare_res != 0)
 {
     my ($stdout, $stderr) =
-      run_command([ 'diff', $dump1_filtered, $dump2_filtered ]);
+      run_command([ 'diff', '-u', $dump1_filtered, $dump2_filtered ]);
     print "=== diff of $dump1_filtered and $dump2_filtered\n";
     print "=== stdout ===\n";
     print $stdout;
diff --git a/src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm b/src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm
new file mode 100644
index 0000000000..7b4a19be2a
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm
@@ -0,0 +1,524 @@
+
+# Copyright (c) 2023, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustUpgrade - helper module for cross-version upgrade tests
+
+=head1 SYNOPSIS
+
+  use PostgreSQL::Test::AdjustUpgrade;
+
+  # Build commands to adjust contents of old-version database before dumping
+  $statements = adjust_database_contents($old_version, %dbnames);
+
+  # Adjust contents of old pg_dumpall output file to match newer version
+  $dump = adjust_old_dumpfile($old_version, $dump);
+
+  # Adjust contents of new pg_dumpall output file to match older version
+  $dump = adjust_new_dumpfile($old_version, $dump);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustUpgrade> encapsulates various hacks needed to
+compare the results of cross-version upgrade tests.
+
+=cut
+
+package PostgreSQL::Test::AdjustUpgrade;
+
+use strict;
+use warnings;
+
+use Exporter 'import';
+use PostgreSQL::Version;
+
+our @EXPORT = qw(
+  adjust_database_contents
+  adjust_old_dumpfile
+  adjust_new_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $statements = adjust_database_contents($old_version, %dbnames)
+
+Generate SQL commands to perform any changes to an old-version installation
+that are needed before we can pg_upgrade it into the current PostgreSQL
+version.
+
+Typically this involves dropping or adjusting no-longer-supported objects.
+
+Arguments:
+
+=over
+
+=item C<old_version>: Branch we are upgrading from, represented as a
+PostgreSQL::Version object.
+
+=item C<dbnames>: Hash of database names present in the old installation.
+
+=back
+
+Returns a reference to a hash, wherein the keys are database names and the
+values are arrayrefs to lists of statements to be run in those databases.
+
+=cut
+
+sub adjust_database_contents
+{
+    my ($old_version, %dbnames) = @_;
+    my $result = {};
+
+    # remove dbs of modules known to cause pg_upgrade to fail
+    # anything not builtin and incompatible should clean up its own db
+    foreach my $bad_module ('test_ddl_deparse', 'tsearch2')
+    {
+        if ($dbnames{"contrib_regression_$bad_module"})
+        {
+            _add_st($result, 'postgres',
+                "drop database contrib_regression_$bad_module");
+            delete($dbnames{"contrib_regression_$bad_module"});
+        }
+    }
+
+    # avoid version number issues with test_ext7
+    if ($dbnames{contrib_regression_test_extensions})
+    {
+        _add_st(
+            $result,
+            'contrib_regression_test_extensions',
+            'drop extension if exists test_ext7');
+    }
+
+    # stuff not supported from release 16
+    if ($old_version >= 12 && $old_version < 16)
+    {
+        # Can't upgrade aclitem in user tables from pre 16 to 16+.
+        _add_st($result, 'regression',
+            'alter table public.tab_core_types drop column aclitem');
+        # Can't handle child tables with locally-generated columns.
+        _add_st(
+            $result, 'regression',
+            'drop table public.gtest_normal_child',
+            'drop table public.gtest_normal_child2');
+    }
+
+    # stuff not supported from release 14
+    if ($old_version < 14)
+    {
+        # postfix operators (some don't exist in very old versions)
+        _add_st(
+            $result,
+            'regression',
+            'drop operator #@# (bigint,NONE)',
+            'drop operator #%# (bigint,NONE)',
+            'drop operator if exists !=- (bigint,NONE)',
+            'drop operator if exists #@%# (bigint,NONE)');
+
+        # get rid of dblink's dependencies on regress.so
+        my $regrdb =
+          $old_version le '9.4'
+          ? 'contrib_regression'
+          : 'contrib_regression_dblink';
+
+        if ($dbnames{$regrdb})
+        {
+            _add_st(
+                $result, $regrdb,
+                'drop function if exists public.putenv(text)',
+                'drop function if exists public.wait_pid(integer)');
+        }
+    }
+
+    # user table OIDs are gone from release 12 on
+    if ($old_version < 12)
+    {
+        my $nooid_stmt = q{
+           DO $stmt$
+           DECLARE
+              rec text;
+           BEGIN
+              FOR rec in
+                 select oid::regclass::text
+                 from pg_class
+                 where relname !~ '^pg_'
+                    and relhasoids
+                    and relkind in ('r','m')
+                 order by 1
+              LOOP
+                 execute 'ALTER TABLE ' || rec || ' SET WITHOUT OIDS';
+                 RAISE NOTICE 'removing oids from table %', rec;
+              END LOOP;
+           END; $stmt$;
+        };
+
+        foreach my $oiddb ('regression', 'contrib_regression_btree_gist')
+        {
+            next unless $dbnames{$oiddb};
+            _add_st($result, $oiddb, $nooid_stmt);
+        }
+
+        # this table had OIDs too, but we'll just drop it
+        if ($old_version >= 10 && $dbnames{'contrib_regression_postgres_fdw'})
+        {
+            _add_st(
+                $result,
+                'contrib_regression_postgres_fdw',
+                'drop foreign table ft_pg_type');
+        }
+    }
+
+    # abstime+friends are gone from release 12 on; but these tables
+    # might or might not be present depending on regression test vintage
+    if ($old_version < 12)
+    {
+        _add_st($result, 'regression',
+            'drop table if exists abstime_tbl, reltime_tbl, tinterval_tbl');
+    }
+
+    # some regression functions gone from release 11 on
+    if ($old_version < 11)
+    {
+        _add_st(
+            $result, 'regression',
+            'drop function if exists public.boxarea(box)',
+            'drop function if exists public.funny_dup17()');
+    }
+
+    # version-0 C functions are no longer supported
+    if ($old_version < 10)
+    {
+        _add_st($result, 'regression',
+            'drop function oldstyle_length(integer, text)');
+    }
+
+    if ($old_version lt '9.5')
+    {
+        # cope with changes of underlying functions
+        _add_st(
+            $result,
+            'regression',
+            'drop operator @#@ (NONE, bigint)',
+            'CREATE OPERATOR @#@ ('
+              . 'PROCEDURE = factorial, RIGHTARG = bigint )',
+            'drop aggregate public.array_cat_accum(anyarray)',
+            'CREATE AGGREGATE array_larger_accum (anyarray) ' . ' ( '
+              . '   sfunc = array_larger, '
+              . '   stype = anyarray, '
+              . '   initcond = $${}$$ ' . '  ) ');
+
+        # "=>" is no longer valid as an operator name
+        _add_st($result, 'regression',
+            'drop operator if exists public.=> (bigint, NONE)');
+    }
+
+    return $result;
+}
+
+# Internal subroutine to add statement(s) to the list for the given db.
+sub _add_st
+{
+    my ($result, $db, @st) = @_;
+
+    $result->{$db} ||= [];
+    push(@{ $result->{$db} }, @st);
+}
+
+=pod
+
+=item adjust_old_dumpfile($old_version, $dump)
+
+Edit a dump output file, taken from the adjusted old-version installation
+by current-version C<pg_dumpall -s>, so that it will match the results of
+C<pg_dumpall -s> on the pg_upgrade'd installation.
+
+Typically this involves coping with cosmetic differences in the output
+of backend subroutines used by pg_dump.
+
+Arguments:
+
+=over
+
+=item C<old_version>: Branch we are upgrading from, represented as a
+PostgreSQL::Version object.
+
+=item C<dump>: Contents of dump file
+
+=back
+
+Returns the modified dump text.
+
+=cut
+
+sub adjust_old_dumpfile
+{
+    my ($old_version, $dump) = @_;
+
+    # use Unix newlines
+    $dump =~ s/\r\n/\n/g;
+
+    # Version comments will certainly not match.
+    $dump =~ s/^-- Dumped from database version.*\n//mg;
+
+    if ($old_version >= 14 && $old_version < 16)
+    {
+        # Fix up some privilege-set discrepancies.
+        $dump =~
+          s {^REVOKE SELECT,INSERT,REFERENCES,DELETE,TRIGGER,TRUNCATE,UPDATE ON TABLE}
+            {REVOKE ALL ON TABLE}mg;
+        $dump =~
+          s {^(GRANT SELECT,INSERT,REFERENCES,TRIGGER,TRUNCATE),UPDATE ON TABLE}
+            {$1,MAINTAIN,UPDATE ON TABLE}mg;
+    }
+
+    if ($old_version < 14)
+    {
+        # Remove mentions of extended hash functions.
+        $dump =~ s {^(\s+OPERATOR\s1\s=\(integer,integer\))\s,\n
+                    \s+FUNCTION\s2\s\(integer,\sinteger\)\spublic\.part_hashint4_noop\(integer,bigint\);}
+                   {$1;}mxg;
+        $dump =~ s {^(\s+OPERATOR\s1\s=\(text,text\))\s,\n
+                    \s+FUNCTION\s2\s\(text,\stext\)\spublic\.part_hashtext_length\(text,bigint\);}
+                   {$1;}mxg;
+    }
+
+    # Change trigger definitions to say ... EXECUTE FUNCTION ...
+    if ($old_version < 12)
+    {
+        # would like to use lookbehind here but perl complains
+        # so do it this way
+        $dump =~ s/
+            (^CREATE\sTRIGGER\s.*?)
+            \sEXECUTE\sPROCEDURE
+            /$1 EXECUTE FUNCTION/mgx;
+    }
+
+    if ($old_version lt '9.6')
+    {
+        # adjust some places where we don't print so many parens anymore
+
+        my $prefix =
+          "'New York'\tnew & york | big & apple | nyc\t'new' & 'york'\t";
+        my $orig = "( 'new' & 'york' | 'big' & 'appl' ) | 'nyc'";
+        my $repl = "'new' & 'york' | 'big' & 'appl' | 'nyc'";
+        $dump =~ s/(?<=^\Q$prefix\E)\Q$orig\E/$repl/mg;
+
+        $prefix =
+          "'Sanct Peter'\tPeterburg | peter | 'Sanct Peterburg'\t'sanct' & 'peter'\t";
+        $orig = "( 'peterburg' | 'peter' ) | 'sanct' & 'peterburg'";
+        $repl = "'peterburg' | 'peter' | 'sanct' & 'peterburg'";
+        $dump =~ s/(?<=^\Q$prefix\E)\Q$orig\E/$repl/mg;
+    }
+
+    if ($old_version lt '9.5')
+    {
+        # adjust some places where we don't print so many parens anymore
+
+        my $prefix = "CONSTRAINT (?:sequence|copy)_con CHECK [(][(]";
+        my $orig   = "((x > 3) AND (y <> 'check failed'::text))";
+        my $repl   = "(x > 3) AND (y <> 'check failed'::text)";
+        $dump =~ s/($prefix)\Q$orig\E/$1$repl/mg;
+
+        $prefix = "CONSTRAINT insert_con CHECK [(][(]";
+        $orig   = "((x >= 3) AND (y <> 'check failed'::text))";
+        $repl   = "(x >= 3) AND (y <> 'check failed'::text)";
+        $dump =~ s/($prefix)\Q$orig\E/$1$repl/mg;
+
+        $orig = "DEFAULT ((-1) * currval('public.insert_seq'::regclass))";
+        $repl =
+          "DEFAULT ('-1'::integer * currval('public.insert_seq'::regclass))";
+        $dump =~ s/\Q$orig\E/$repl/mg;
+
+        my $expr =
+          "(rsl.sl_color = rsh.slcolor) AND (rsl.sl_len_cm >= rsh.slminlen_cm)";
+        $dump =~ s/WHERE \(\(\Q$expr\E\)/WHERE ($expr/g;
+
+        $expr =
+          "(rule_and_refint_t3.id3a = new.id3a) AND (rule_and_refint_t3.id3b = new.id3b)";
+        $dump =~ s/WHERE \(\(\Q$expr\E\)/WHERE ($expr/g;
+
+        $expr =
+          "(rule_and_refint_t3_1.id3a = new.id3a) AND (rule_and_refint_t3_1.id3b = new.id3b)";
+        $dump =~ s/WHERE \(\(\Q$expr\E\)/WHERE ($expr/g;
+    }
+
+    if ($old_version lt '9.3')
+    {
+        # CREATE VIEW/RULE statements were not pretty-printed before 9.3.
+        # To cope, reduce all whitespace sequences within them to one space.
+        # This must be done on both old and new dumps.
+        $dump = _mash_view_whitespace($dump);
+
+        # _mash_view_whitespace doesn't handle multi-command rules;
+        # rather than trying to fix that, just hack the exceptions manually.
+
+        my $prefix =
+          "CREATE RULE rtest_sys_del AS ON DELETE TO public.rtest_system DO (DELETE FROM public.rtest_interface WHERE
(rtest_interface.sysname= old.sysname);"; 
+        my $line2 = " DELETE FROM public.rtest_admin";
+        my $line3 = " WHERE (rtest_admin.sysname = old.sysname);";
+        $dump =~
+          s/(?<=\Q$prefix\E)\Q$line2$line3\E \);/\n$line2\n $line3\n);/mg;
+
+        $prefix =
+          "CREATE RULE rtest_sys_upd AS ON UPDATE TO public.rtest_system DO (UPDATE public.rtest_interface SET sysname
=new.sysname WHERE (rtest_interface.sysname = old.sysname);"; 
+        $line2 = " UPDATE public.rtest_admin SET sysname = new.sysname";
+        $line3 = " WHERE (rtest_admin.sysname = old.sysname);";
+        $dump =~
+          s/(?<=\Q$prefix\E)\Q$line2$line3\E \);/\n$line2\n $line3\n);/mg;
+
+        # and there's one place where pre-9.3 uses a different table alias
+        $dump =~ s {^(CREATE\sRULE\srule_and_refint_t3_ins\sAS\s
+             ON\sINSERT\sTO\spublic\.rule_and_refint_t3\s
+             WHERE\s\(EXISTS\s\(SELECT\s1\sFROM\spublic\.rule_and_refint_t3)\s
+             (WHERE\s\(\(rule_and_refint_t3)
+             (\.id3a\s=\snew\.id3a\)\sAND\s\(rule_and_refint_t3)
+             (\.id3b\s=\snew\.id3b\)\sAND\s\(rule_and_refint_t3)}
+        {$1 rule_and_refint_t3_1 $2_1$3_1$4_1}mx;
+
+        # Also fix old use of NATURAL JOIN syntax
+        $dump =~ s {NATURAL JOIN public\.credit_card r}
+            {JOIN public.credit_card r USING (cid)}mg;
+        $dump =~ s {NATURAL JOIN public\.credit_usage r}
+            {JOIN public.credit_usage r USING (cid)}mg;
+    }
+
+    # Suppress blank lines, as some places in pg_dump emit more or fewer.
+    $dump =~ s/\n\n+/\n/g;
+
+    return $dump;
+}
+
+# Internal subroutine to mangle whitespace within view/rule commands.
+# Any consecutive sequence of whitespace is reduced to one space.
+sub _mash_view_whitespace
+{
+    my ($dump) = @_;
+
+    foreach my $leader ('CREATE VIEW', 'CREATE RULE')
+    {
+        my @splitchunks = split $leader, $dump;
+
+        $dump = shift(@splitchunks);
+        foreach my $chunk (@splitchunks)
+        {
+            my @thischunks = split /;/, $chunk, 2;
+            my $stmt = shift(@thischunks);
+
+            # now $stmt is just the body of the CREATE VIEW/RULE
+            $stmt =~ s/\s+/ /sg;
+            # we also need to smash these forms for sub-selects and rules
+            $stmt =~ s/\( SELECT/(SELECT/g;
+            $stmt =~ s/\( INSERT/(INSERT/g;
+            $stmt =~ s/\( UPDATE/(UPDATE/g;
+            $stmt =~ s/\( DELETE/(DELETE/g;
+
+            $dump .= $leader . $stmt . ';' . $thischunks[0];
+        }
+    }
+    return $dump;
+}
+
+=pod
+
+=item adjust_new_dumpfile($old_version, $dump)
+
+Edit a dump output file, taken from the pg_upgrade'd installation
+by current-version C<pg_dumpall -s>, so that it will match the old
+dump output file as adjusted by C<adjust_old_dumpfile>.
+
+Typically this involves deleting data not present in the old installation.
+
+Arguments:
+
+=over
+
+=item C<old_version>: Branch we are upgrading from, represented as a
+PostgreSQL::Version object.
+
+=item C<dump>: Contents of dump file
+
+=back
+
+Returns the modified dump text.
+
+=cut
+
+sub adjust_new_dumpfile
+{
+    my ($old_version, $dump) = @_;
+
+    # use Unix newlines
+    $dump =~ s/\r\n/\n/g;
+
+    # Version comments will certainly not match.
+    $dump =~ s/^-- Dumped from database version.*\n//mg;
+
+    if ($old_version < 14)
+    {
+        # Suppress noise-word uses of IN in CREATE/ALTER PROCEDURE.
+        $dump =~ s/^(CREATE PROCEDURE .*?)\(IN /$1(/mg;
+        $dump =~ s/^(ALTER PROCEDURE .*?)\(IN /$1(/mg;
+        $dump =~ s/^(CREATE PROCEDURE .*?), IN /$1, /mg;
+        $dump =~ s/^(ALTER PROCEDURE .*?), IN /$1, /mg;
+        $dump =~ s/^(CREATE PROCEDURE .*?), IN /$1, /mg;
+        $dump =~ s/^(ALTER PROCEDURE .*?), IN /$1, /mg;
+
+        # Remove SUBSCRIPT clauses in CREATE TYPE.
+        $dump =~ s/^\s+SUBSCRIPT = raw_array_subscript_handler,\n//mg;
+
+        # Remove multirange_type_name clauses in CREATE TYPE AS RANGE.
+        $dump =~ s {,\n\s+multirange_type_name = .*?(,?)$} {$1}mg;
+
+        # Remove mentions of extended hash functions.
+        $dump =~
+          s {^ALTER\sOPERATOR\sFAMILY\spublic\.part_test_int4_ops\sUSING\shash\sADD\n
+                        \s+FUNCTION\s2\s\(integer,\sinteger\)\spublic\.part_hashint4_noop\(integer,bigint\);} {}mxg;
+        $dump =~
+          s {^ALTER\sOPERATOR\sFAMILY\spublic\.part_test_text_ops\sUSING\shash\sADD\n
+                        \s+FUNCTION\s2\s\(text,\stext\)\spublic\.part_hashtext_length\(text,bigint\);} {}mxg;
+    }
+
+    # pre-v12 dumps will not say anything about default_table_access_method.
+    if ($old_version < 12)
+    {
+        $dump =~ s/^SET default_table_access_method = heap;\n//mg;
+    }
+
+    # dumps from pre-9.6 dblink may include redundant ACL settings
+    if ($old_version lt '9.6')
+    {
+        my $comment =
+          "-- Name: FUNCTION dblink_connect_u\(.*?\); Type: ACL; Schema: public; Owner: .*";
+        my $sql =
+          "REVOKE ALL ON FUNCTION public\.dblink_connect_u\(.*?\) FROM PUBLIC;";
+        $dump =~ s/^--\n$comment\n--\n+$sql\n+//mg;
+    }
+
+    if ($old_version lt '9.3')
+    {
+        # CREATE VIEW/RULE statements were not pretty-printed before 9.3.
+        # To cope, reduce all whitespace sequences within them to one space.
+        # This must be done on both old and new dumps.
+        $dump = _mash_view_whitespace($dump);
+    }
+
+    # Suppress blank lines, as some places in pg_dump emit more or fewer.
+    $dump =~ s/\n\n+/\n/g;
+
+    return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff -pudr client-code-REL_16.orig/PGBuild/Modules/TestUpgradeXversion.pm
client-code-REL_16/PGBuild/Modules/TestUpgradeXversion.pm
--- client-code-REL_16.orig/PGBuild/Modules/TestUpgradeXversion.pm    2023-01-13 12:20:51.000000000 -0500
+++ client-code-REL_16/PGBuild/Modules/TestUpgradeXversion.pm    2023-01-16 15:01:22.502366802 -0500
@@ -323,31 +323,6 @@ sub save_for_testing
         return if $?;
     }

-    if ($this_branch ne 'HEAD' && $this_branch le 'REL9_4_STABLE')
-    {
-        my $opsql = 'drop operator if exists public.=> (bigint, NONE)';
-
-        # syntax is illegal in 9.5 and later, and it shouldn't
-        # be possible for it to exist there anyway.
-        # quoting the operator can also fail,  so it's left unquoted.
-        run_psql("$installdir/bin/psql", "-e", $opsql, "regression",
-            "$upgrade_loc/fix.log", 1);
-        return if $?;
-    }
-
-    # remove dbs of modules known to cause pg_upgrade to fail
-    # anything not builtin and incompatible should clean up its own db
-    # e.g. jsonb_set_lax
-
-    foreach my $bad_module ("test_ddl_deparse")
-    {
-        my $dsql = "drop database if exists contrib_regression_$bad_module";
-
-        run_psql("$installdir/bin/psql", "-e", $dsql,
-            "postgres", "$upgrade_loc/fix.log", 1);
-        return if $?;
-    }
-
     # use a different logfile here to get around windows sharing issue
     system( qq{"$installdir/bin/pg_ctl" -D "$installdir/data-C" -w stop }
           . qq{>> "$upgrade_loc/ctl2.log" 2>&1});
@@ -375,6 +350,21 @@ sub test_upgrade    ## no critic (Subrou
     print time_str(), "checking upgrade from $oversion to $this_branch ...\n"
       if $verbose;

+    # load helper module from source tree
+    unshift(@INC, "$self->{pgsql}/src/test/perl");
+    require PostgreSQL::Test::AdjustUpgrade;
+    PostgreSQL::Test::AdjustUpgrade->import;
+    shift(@INC);
+
+    # if $oversion isn't HEAD, convert it into a PostgreSQL::Version object
+    my $old_version = $oversion;
+    if ($old_version ne 'HEAD')
+    {
+        $old_version =~ s/REL_?(\d+(?:_\d+)?)_STABLE/$1/;
+        $old_version =~ s/_/./;
+        $old_version = PostgreSQL::Version->new($old_version);
+    }
+
     rmtree "$other_branch/inst/$upgrade_test";
     copydir(
         "$other_branch/inst/data-C",
@@ -414,6 +404,7 @@ sub test_upgrade    ## no critic (Subrou

     return if $?;

+    # collect names of databases present in old installation.
     my $sql = 'select datname from pg_database';

     run_psql("psql", "-A -t", $sql, "postgres",
@@ -425,186 +416,22 @@ sub test_upgrade    ## no critic (Subrou
     do { s/\r$//; $dbnames{$_} = 1; }
       foreach @dbnames;

-    if ($this_branch gt 'REL9_6_STABLE' || $this_branch eq 'HEAD')
-    {
-        run_psql(
-            "$other_branch/inst/bin/psql",                         "-e",
-            "drop database if exists contrib_regression_tsearch2", "postgres",
-            "$upgrade_loc/$oversion-copy.log",                     1
-        );
-        return if $?;
-
-        run_psql(
-            "$other_branch/inst/bin/psql",
-            "-e",
-            "drop function if exists oldstyle_length(integer, text)",
-            "regression",
-            "$upgrade_loc/$oversion-copy.log",
-            1
-        );
-        return if $?;
-    }
-
-    # some regression functions gone from release 11 on
-    if (   ($this_branch ge 'REL_11_STABLE' || $this_branch eq 'HEAD')
-        && ($oversion lt 'REL_11_STABLE' && $oversion ne 'HEAD'))
-    {
-        my $missing_funcs = q{drop function if exists public.boxarea(box);
-                              drop function if exists public.funny_dup17();
-                            };
-        $missing_funcs =~ s/\n//g;
-
-        run_psql("$other_branch/inst/bin/psql", "-e", $missing_funcs,
-            "regression", "$upgrade_loc/$oversion-copy.log", 1);
-        return if $?;
-    }
-
-    # avoid version number issues with test_ext7
-    if ($dbnames{contrib_regression_test_extensions})
-    {
-        my $noext7 = "drop extension if exists test_ext7";
-        run_psql(
-            "$other_branch/inst/bin/psql", "-e", $noext7,
-            "contrib_regression_test_extensions",
-            "$upgrade_loc/$oversion-copy.log", 1
-        );
-        return if $?;
-    }
-
-    # user table OIDS and abstime+friends are gone from release 12 on
-    if (   ($this_branch gt 'REL_11_STABLE' || $this_branch eq 'HEAD')
-        && ($oversion le 'REL_11_STABLE' && $oversion ne 'HEAD'))
-    {
-        my $nooid_stmt = q{
-           DO $stmt$
-           DECLARE
-              rec text;
-           BEGIN
-              FOR rec in
-                 select oid::regclass::text
-                 from pg_class
-                 where relname !~ '^pg_'
-                    and relhasoids
-                    and relkind in ('r','m')
-                 order by 1
-              LOOP
-                 execute 'ALTER TABLE ' || rec || ' SET WITHOUT OIDS';
-                 RAISE NOTICE 'removing oids from table %', rec;
-              END LOOP;
-           END; $stmt$;
-        };
-        foreach my $oiddb ("regression", "contrib_regression_btree_gist")
-        {
-            next unless $dbnames{$oiddb};
-            run_psql("$other_branch/inst/bin/psql", "-e", $nooid_stmt,
-                "$oiddb", "$upgrade_loc/$oversion-copy.log", 1);
-            return if $?;
-        }
-
-        if (   $oversion ge 'REL_10_STABLE'
-            && $dbnames{'contrib_regression_postgres_fdw'})
-        {
-            run_psql(
-                "$other_branch/inst/bin/psql",
-                "-e",
-                "drop foreign table if exists ft_pg_type",
-                "contrib_regression_postgres_fdw",
-                "$upgrade_loc/$oversion-copy.log",
-                1
-            );
-            return if $?;
-        }
-
-        if ($oversion lt 'REL9_3_STABLE')
-        {
-            run_psql(
-                "$other_branch/inst/bin/psql",
-                "-e",
-                "drop table if exists abstime_tbl, reltime_tbl, tinterval_tbl",
-                "regression",
-                "$upgrade_loc/$oversion-copy.log",
-                1
-            );
-            return if $?;
-        }
-    }
-
-    # stuff not supported from release 14
-    if (   ($this_branch gt 'REL_13_STABLE' || $this_branch eq 'HEAD')
-        && ($oversion le 'REL_13_STABLE' && $oversion ne 'HEAD'))
+    if ($oversion ne $this_branch)
     {
-        my $prstmt = join(';',
-            'drop operator if exists #@# (bigint,NONE)',
-            'drop operator if exists #%# (bigint,NONE)',
-            'drop operator if exists !=- (bigint,NONE)',
-            'drop operator if exists #@%# (bigint,NONE)');
-
-        run_psql("$other_branch/inst/bin/psql", "-e", $prstmt,
-            "regression", "$upgrade_loc/$oversion-copy.log", 1);
-        return if $?;
-
-        $prstmt = "drop function if exists public.putenv(text)";
-
-        my $regrdb =
-          $oversion le "REL9_4_STABLE"
-          ? "contrib_regression"
-          : "contrib_regression_dblink";
-
-        if ($dbnames{$regrdb})
-        {
-            run_psql("$other_branch/inst/bin/psql", "-e", $prstmt,
-                "$regrdb", "$upgrade_loc/$oversion-copy.log", 1);
-            return if $?;
-        }
+        # obtain and execute commands needed to make old database upgradable.
+        my $adjust_cmds = adjust_database_contents($old_version, %dbnames);

-        if ($oversion le 'REL9_4_STABLE')
+        foreach my $updb (keys %$adjust_cmds)
         {
-            # this is fixed in 9.5 and later
-            $prstmt = join(';',
-                'drop operator @#@ (NONE, bigint)',
-                'CREATE OPERATOR @#@ ('
-                  . 'PROCEDURE = factorial, '
-                  . 'RIGHTARG = bigint )');
-            run_psql("$other_branch/inst/bin/psql", "-e", $prstmt,
-                "regression", "$upgrade_loc/$oversion-copy.log", 1);
-            return if $?;
-        }
+            my $upcmds = join(";\n", @{ $adjust_cmds->{$updb} });

-        if ($oversion le 'REL9_4_STABLE')
-        {
-            # this is fixed in 9.5 and later
-            $prstmt = join(';',
-                'drop aggregate if exists public.array_cat_accum(anyarray)',
-                'CREATE AGGREGATE array_larger_accum (anyarray) ' . ' ( '
-                  . '   sfunc = array_larger, '
-                  . '   stype = anyarray, '
-                  . '   initcond = $${}$$ '
-                  . '  ) ');
-            run_psql("$other_branch/inst/bin/psql", "-e", $prstmt,
-                "regression", "$upgrade_loc/$oversion-copy.log", 1);
+            run_psql("$other_branch/inst/bin/psql", "-e -v ON_ERROR_STOP=1",
+                $upcmds, $updb, "$upgrade_loc/$oversion-fix.log", 1);
             return if $?;
         }
     }

-    # stuff not supported from release 16
-    if (   ($this_branch gt 'REL_15_STABLE' || $this_branch eq 'HEAD')
-        && ($oversion le 'REL_15_STABLE' && $oversion ne 'HEAD'))
-    {
-        # Can't upgrade aclitem in user tables from pre 16 to 16+.
-        # Also can't handle child tables with newly-generated columns.
-        my $prstmt = join(
-            ';',
-            'alter table if exists public.tab_core_types
-                          drop column if exists aclitem',
-            'drop table if exists public.gtest_normal_child',
-            'drop table if exists public.gtest_normal_child2'
-        );
-
-        run_psql("$other_branch/inst/bin/psql", "-e", $prstmt,
-            "regression", "$upgrade_loc/$oversion-copy.log", 1);
-        return if $?;
-    }
-
+    # perform a dump from the old database for comparison purposes.
     my $extra_digits = "";

     if (   $oversion ne 'HEAD'
@@ -793,65 +620,40 @@ sub test_upgrade    ## no critic (Subrou
         return if $?;
     }

-    foreach my $dump ("$upgrade_loc/origin-$oversion.sql",
-        "$upgrade_loc/converted-$oversion-to-$this_branch.sql")
-    {
-        # Change trigger definitions to say ... EXECUTE FUNCTION ...
+    # Slurp the pg_dump output files, and filter them if not same version.
+    my $olddumpfile = "$upgrade_loc/origin-$oversion.sql";
+    my $olddump     = file_contents($olddumpfile);

-        my $contents = file_contents($dump);
+    $olddump = adjust_old_dumpfile($old_version, $olddump)
+      if ($oversion ne $this_branch);

-        # would like to use lookbehind here but perl complains
-        # so do it this way
-        $contents =~ s/
-                         (^CREATE\sTRIGGER\s.*?)
-                         \sEXECUTE\sPROCEDURE
-                      /$1 EXECUTE FUNCTION/mgx;
-        open(my $dh, '>', "$dump.fixed") || die "opening $dump.fixed";
-        print $dh $contents;
-        close($dh);
-    }
+    my $newdumpfile = "$upgrade_loc/converted-$oversion-to-$this_branch.sql";
+    my $newdump     = file_contents($newdumpfile);

-    system( qq{diff -I "^\$" -I "SET default_table_access_method = heap;" }
-          . qq{ -I "^SET default_toast_compression = 'pglz';\$" -I "^-- " }
-          . qq{-u "$upgrade_loc/origin-$oversion.sql.fixed" }
-          . qq{"$upgrade_loc/converted-$oversion-to-$this_branch.sql.fixed" }
-          . qq{> "$upgrade_loc/dumpdiff-$oversion" 2>&1});
+    $newdump = adjust_new_dumpfile($old_version, $newdump)
+      if ($oversion ne $this_branch);

-    # diff exits with status 1 if files differ
-    return if $? >> 8 > 1;
+    # Always write out the filtered files, to aid in diagnosing filter bugs.
+    open(my $odh, '>', "$olddumpfile.fixed")
+      || die "opening $olddumpfile.fixed: $!";
+    print $odh $olddump;
+    close($odh);
+    open(my $ndh, '>', "$newdumpfile.fixed")
+      || die "opening $newdumpfile.fixed: $!";
+    print $ndh $newdump;
+    close($ndh);

-    open(my $diffile, '<', "$upgrade_loc/dumpdiff-$oversion")
-      || die "opening $upgrade_loc/dumpdiff-$oversion: $!";
-    my $difflines = 0;
-    while (<$diffile>)
+    # Are the results the same?
+    if ($olddump ne $newdump)
     {
-        $difflines++ if /^[+-]/;
-    }
-    close($diffile);
-
-    # If the versions match we require that there be no diff lines.
-    # In the past we have seen a handful of diffs from reordering of
-    # large object output, but that appears to have disppeared.
-    # If the versions don't match we heuristically allow more lines of diffs
-    # based on observed differences. For versions from 9.6 on, that's
-    # not very many lines, though.
+        # Trouble, so run diff to show the problem.
+        system( qq{diff -u "$olddumpfile.fixed" "$newdumpfile.fixed" }
+              . qq{> "$upgrade_loc/dumpdiff-$oversion" 2>&1});

-    if (
-        ($oversion eq $this_branch && $difflines == 0)
-        || (   $oversion ne $this_branch
-            && $oversion ge 'REL9_6_STABLE'
-            && $difflines < 90)
-        || (   $oversion ne $this_branch
-            && $oversion lt 'REL9_6_STABLE'
-            && $difflines < 700)
-      )
-    {
-        return 1;
-    }
-    else
-    {
         return;
     }
+
+    return 1;
 }

 sub installcheck