Thread: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

Hi,
(Adding Andrew in CC for the buildfarm and PostgresNode parts.)

$subject has been around for a couple of years now, with the following
threads:
https://www.postgresql.org/message-id/20180126080026.GI17847@paquier.xyz
https://www.postgresql.org/message-id/CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com

An advantage of moving to TAP is that we can then remove the support
for upgrades within the MSVC scripts, and also remove pg_upgrade's
test.sh that has accumulated tweaks that are solved by the TAP tests,
resulting in cleanup:
 8 files changed, 230 insertions(+), 403 deletions(-)

Based on the past discussions, there were two obstacles preventing to
do this switch:
- Support for tests with older versions, something where the gap as
been closed thanks to Andrew's work in 4c4eaf3d.
- Buildfarm support, and I am not sure how things need to be extended
there.

Another thing to note is that HEAD uses oldbindir, bindir and libdir
to track the location of the old and new libraries and binaries.  With
the infrastructure in place, once can define only an install path for
a PostgresNode, so this version uses only two variables:
- oldinstall, for the installation path of the version to upgrade
from.
- oldsrc, to point to the source of the old version.

It is not difficult to switch to one approach or the other, but
reducing the logic to a minimum number of variables is a good deal to
take IMO.

I have been testing this patch a bit with older versions, down to 12,
and that was logically working, and PostgresNode may need more to be
able to work with ~11, as documented in get_new_node().  And I have
not tested that with MSVC yet.  Anyway, attached is a new patch to
make the discussion move on.  Even if there is still work to be done
here, would people here still support this switch?

Thanks,
--
Michael

On 5/14/21 10:26 PM, Michael Paquier wrote:
> Hi,
> (Adding Andrew in CC for the buildfarm and PostgresNode parts.)
>
> $subject has been around for a couple of years now, with the following
> threads:
> https://www.postgresql.org/message-id/20180126080026.GI17847@paquier.xyz
> https://www.postgresql.org/message-id/CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com
>
> An advantage of moving to TAP is that we can then remove the support
> for upgrades within the MSVC scripts, and also remove pg_upgrade's
> test.sh that has accumulated tweaks that are solved by the TAP tests,
> resulting in cleanup:
>  8 files changed, 230 insertions(+), 403 deletions(-)
>
> Based on the past discussions, there were two obstacles preventing to
> do this switch:
> - Support for tests with older versions, something where the gap as
> been closed thanks to Andrew's work in 4c4eaf3d.
> - Buildfarm support, and I am not sure how things need to be extended
> there.
>
> Another thing to note is that HEAD uses oldbindir, bindir and libdir
> to track the location of the old and new libraries and binaries.  With
> the infrastructure in place, once can define only an install path for
> a PostgresNode, so this version uses only two variables:
> - oldinstall, for the installation path of the version to upgrade
> from.
> - oldsrc, to point to the source of the old version.
>
> It is not difficult to switch to one approach or the other, but
> reducing the logic to a minimum number of variables is a good deal to
> take IMO.
>
> I have been testing this patch a bit with older versions, down to 12,
> and that was logically working, and PostgresNode may need more to be
> able to work with ~11, as documented in get_new_node().  And I have
> not tested that with MSVC yet.  Anyway, attached is a new patch to
> make the discussion move on.  Even if there is still work to be done
> here, would people here still support this switch?



PostgresNode is currently able to create nodes suitable for upgrade down
to release 10. If we add '-w' to the 'pg_ctl start' flags that can
extend down to release 9.5. (Just tested) I think we should do that
forthwith. '-w' is now the default, but having it there explicitly does
no harm.


If people are interested in what's incompatible on older versions, they
can look at
<https://gitlab.com/adunstan/postgresnodeng/-/blob/master/PostgresNode.pm>
starting at about line 2764.


I don't think this will work, though, unless there is enough data to
exercise pg_upgrade fairly thoroughly. The approach taken by both
test.sh and (somewhat more comprehensively) by the buildfarm cross
version upgrade module is to test a cluster where the regression tests
have been run. That might be more difficult when testing against older
versions, so I have published a snapshot of the dumps of each of the
versions we tests against in the buildfarm animal crake. These could be
loaded into PostgresNode instances and then an upgrade attempted. See
<https://gitlab.com/adunstan/pg-old-bin/-/tree/master/data>. The data
goes back to 9.2. These compressed dumps are a couple of megabytes each,
not huge.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Sat, May 15, 2021 at 02:22:24PM -0400, Andrew Dunstan wrote:
> PostgresNode is currently able to create nodes suitable for upgrade down
> to release 10. If we add '-w' to the 'pg_ctl start' flags that can
> extend down to release 9.5. (Just tested) I think we should do that
> forthwith. '-w' is now the default, but having it there explicitly does
> no harm.

Agreed.  When testing manually, I have personally never worked on any
patches that required binaries older than 9.4, so I would be fine if
the TAP tests are able to work easily down to 9.5, even if pg_upgrade
is supported down to 8.4.

> If people are interested in what's incompatible on older versions, they
> can look at
> <https://gitlab.com/adunstan/postgresnodeng/-/blob/master/PostgresNode.pm>
> starting at about line 2764.

We should really have adjust_conf() at some point in the in-core
module.

> I don't think this will work, though, unless there is enough data to
> exercise pg_upgrade fairly thoroughly. The approach taken by both
> test.sh and (somewhat more comprehensively) by the buildfarm cross
> version upgrade module is to test a cluster where the regression tests
> have been run.

Yeah, that's what my patch is doing with pg_regress, FWIW.  This
requires regress.so from the old version.

> That might be more difficult when testing against older
> versions, so I have published a snapshot of the dumps of each of the
> versions we tests against in the buildfarm animal crake. These could be
> loaded into PostgresNode instances and then an upgrade attempted. See
> <https://gitlab.com/adunstan/pg-old-bin/-/tree/master/data>. The data
> goes back to 9.2. These compressed dumps are a couple of megabytes each,
> not huge.

I agree that this can be simpler in some cases.  In your experience,
how much of an issue is it when it becomes necessary to keep around
binaries that rely on libraries older than what a system can support?
It is easy to counter issues in this area with OpenSSL and
non-necessary things, but we had in the past also cases where we had
code that conflicted with the kernel, aka 3e68686.

At the end of this exercise, what I think we should achieve is to:
1) Reduce the diff between the buildfarm code and the in-core code.
2) Get rid of test.sh.
3) Be able to run easily upgrade tests across major versions for
developers.

As of now, 3) requires some extra facilities depending on if this is
done by the buildfarm or the in-core tests:
1) Path to the source code of the old version.  This is required once
it becomes necessary to find out src/test/regress/ for the schedule,
the tests to run and its regress.so.  There is no need to do that if
you have a dump of the old instance.
2) Path to a custom dump to replace the run with pg_regress from 1).
3) Location of the old binaries, for pg_upgrade.  When it comes to
PostgresNode, we require an install path, so we cannot use directly
the location of the binaries.

Looking at the code of the buildfarm, its code does something smarter
than what my patch or HEAD's test.sh does now, as these require the
path for the old source.  The buildfarm code first scans for the
probin's used in the regression database and then updates any
references.  What test.sh and my patch do is using the path to the old
source code and run a single UPDATE.  The approach taken by the
buildfarm code is more portable, as a path to the old source code
becomes necessary only if running pg_regress manually.  So, what about
doing the following thing?
1) Update the TAP test so as probin entries are updated in the same way
as the buildfarm.
2) Allow one to specify a path to a custom dump, or a path to the old
source code for pg_regress.

If we do that, then it should be possible to reduce the code footprint
in the buildfarm code, while still allowing people to test major
upgrades in the same old-fashioned way, right?  That's assuming that
PostgresNode is made compatible down to 9.2, of course, as a first
step, as that's the range of the dumps you are keeping around for the
buildfarm.

Thoughts?
--
Michael

On 5/16/21 9:55 PM, Michael Paquier wrote:
> On Sat, May 15, 2021 at 02:22:24PM -0400, Andrew Dunstan wrote:
>> PostgresNode is currently able to create nodes suitable for upgrade down
>> to release 10. If we add '-w' to the 'pg_ctl start' flags that can
>> extend down to release 9.5. (Just tested) I think we should do that
>> forthwith. '-w' is now the default, but having it there explicitly does
>> no harm.
> Agreed.  When testing manually, I have personally never worked on any
> patches that required binaries older than 9.4, so I would be fine if
> the TAP tests are able to work easily down to 9.5, even if pg_upgrade
> is supported down to 8.4.
>
>> If people are interested in what's incompatible on older versions, they
>> can look at
>> <https://gitlab.com/adunstan/postgresnodeng/-/blob/master/PostgresNode.pm>
>> starting at about line 2764.
> We should really have adjust_conf() at some point in the in-core
> module.



Yes, I'm going to be proposing a series of smallish patches including
these when the tree is branched (which I hope will be in a few weeks).


>> I don't think this will work, though, unless there is enough data to
>> exercise pg_upgrade fairly thoroughly. The approach taken by both
>> test.sh and (somewhat more comprehensively) by the buildfarm cross
>> version upgrade module is to test a cluster where the regression tests
>> have been run.
> Yeah, that's what my patch is doing with pg_regress, FWIW.  This
> requires regress.so from the old version.
>
>> That might be more difficult when testing against older
>> versions, so I have published a snapshot of the dumps of each of the
>> versions we tests against in the buildfarm animal crake. These could be
>> loaded into PostgresNode instances and then an upgrade attempted. See
>> <https://gitlab.com/adunstan/pg-old-bin/-/tree/master/data>. The data
>> goes back to 9.2. These compressed dumps are a couple of megabytes each,
>> not huge.
> I agree that this can be simpler in some cases.  In your experience,
> how much of an issue is it when it becomes necessary to keep around
> binaries that rely on libraries older than what a system can support?
> It is easy to counter issues in this area with OpenSSL and
> non-necessary things, but we had in the past also cases where we had
> code that conflicted with the kernel, aka 3e68686.


That one at least isn't an issue. Old versions of postgres didn't have
pg_rewind.



>
> At the end of this exercise, what I think we should achieve is to:
> 1) Reduce the diff between the buildfarm code and the in-core code.
> 2) Get rid of test.sh.
> 3) Be able to run easily upgrade tests across major versions for
> developers.
>
> As of now, 3) requires some extra facilities depending on if this is
> done by the buildfarm or the in-core tests:
> 1) Path to the source code of the old version.  This is required once
> it becomes necessary to find out src/test/regress/ for the schedule,
> the tests to run and its regress.so.  There is no need to do that if
> you have a dump of the old instance.
> 2) Path to a custom dump to replace the run with pg_regress from 1).
> 3) Location of the old binaries, for pg_upgrade.  When it comes to
> PostgresNode, we require an install path, so we cannot use directly
> the location of the binaries.
>
> Looking at the code of the buildfarm, its code does something smarter
> than what my patch or HEAD's test.sh does now, as these require the
> path for the old source.  The buildfarm code first scans for the
> probin's used in the regression database and then updates any
> references.  What test.sh and my patch do is using the path to the old
> source code and run a single UPDATE.  The approach taken by the
> buildfarm code is more portable, as a path to the old source code
> becomes necessary only if running pg_regress manually.  So, what about
> doing the following thing?
> 1) Update the TAP test so as probin entries are updated in the same way
> as the buildfarm.
> 2) Allow one to specify a path to a custom dump, or a path to the old
> source code for pg_regress.
>
> If we do that, then it should be possible to reduce the code footprint
> in the buildfarm code, while still allowing people to test major
> upgrades in the same old-fashioned way, right?  That's assuming that
> PostgresNode is made compatible down to 9.2, of course, as a first
> step, as that's the range of the dumps you are keeping around for the
> buildfarm.
>


I'm intending to add some older dumps. -) But for now 9.2 is a good target.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Mon, May 17, 2021 at 12:32:13PM -0400, Andrew Dunstan wrote:
> On 5/16/21 9:55 PM, Michael Paquier wrote:
> Yes, I'm going to be proposing a series of smallish patches including
> these when the tree is branched (which I hope will be in a few weeks).

Thanks!  That clearly needs to happen first.  I'll help reviewing
these.

>> If we do that, then it should be possible to reduce the code footprint
>> in the buildfarm code, while still allowing people to test major
>> upgrades in the same old-fashioned way, right?  That's assuming that
>> PostgresNode is made compatible down to 9.2, of course, as a first
>> step, as that's the range of the dumps you are keeping around for the
>> buildfarm.
>
> I'm intending to add some older dumps. -) But for now 9.2 is a good target.

Makes sense.  For now, I'll update this patch set so as it is possible
to use custom dumps, as an option in parallel of pg_regress when
specifying a different source code path.  I'll also decouple the
business with probin updates and stick with the approach used by the
buildfarm code.
--
Michael

On Tue, May 18, 2021 at 10:49:39AM +0900, Michael Paquier wrote:
> Makes sense.  For now, I'll update this patch set so as it is possible
> to use custom dumps, as an option in parallel of pg_regress when
> specifying a different source code path.  I'll also decouple the
> business with probin updates and stick with the approach used by the
> buildfarm code.

This has proved to not be that difficult.  With the updated version
attached, pg_upgrade has two modes to set up the old instance used for
the upgrade with older binaries:
- With oldsrc and oldinstall set, pg_regress gets used, same way as
HEAD.
- With olddump and oldinstall set, an old dump is loaded instead in
the old instance before launching the upgrade.

oldsrc and olddump are exclusive options.  Similarly to HEAD, the
dumps taken from the old and new instances generate diffs that can be
inspected manually.  The updates of probin are done without any
dependencies to the source path of the old instance, copying from the
buildfarm.

While on it, I have fixed a couple of things that exist in test.sh but
were not reflected in this new script:
- Handling of postfix operations with ~13 clusters.
- Handling oldstyle_length for ~9.6 clusters.
- Handling of EXTRA_REGRESS_OPT.

This stuff still needs to be expanded depending on how PostgresNode is
made backward-compatible, but I'll wait for that to happen before
going further down here.  I have also spent some time testing all that
with MSVC, and the installation paths used for pg_regress&co make the
script a tad more confusing, so I have dropped this part for now.

Thanks,
--
Michael

On Tue, Sep 07, 2021 at 02:43:15PM -0700, Rachel Heaton wrote:
> Running check tests in the pg_upgrade folder fails for this reason.

Thanks, rebased as attached.  Andrew has posted another patch set that
completely reworks the shape of the modules by moving them into a
dedicated namespace, meaning that this is going to break again.  I'll
see about that when we reach this point.
--
Michael

On Thu, May 20, 2021 at 03:07:56PM +0900, Michael Paquier wrote:
> This stuff still needs to be expanded depending on how PostgresNode is
> made backward-compatible, but I'll wait for that to happen before
> going further down here.  I have also spent some time testing all that
> with MSVC, and the installation paths used for pg_regress&co make the
> script a tad more confusing, so I have dropped this part for now.

Andrew, as this is a bit tied to the buildfarm code and any
simplifications that could happen there, do you have any comments
and/or suggestions for this patch?

This still applies on HEAD and it holds all the properties of the
existing test by using PostgresNodes that point to older installations
for the business with binaries and libraries business.  There is one
part where pg_upgrade logs into src/test/regress/, which is not good,
but that should be easily fixable.
--
Michael

On 10/1/21 2:19 AM, Michael Paquier wrote:
> On Thu, May 20, 2021 at 03:07:56PM +0900, Michael Paquier wrote:
>> This stuff still needs to be expanded depending on how PostgresNode is
>> made backward-compatible, but I'll wait for that to happen before
>> going further down here.  I have also spent some time testing all that
>> with MSVC, and the installation paths used for pg_regress&co make the
>> script a tad more confusing, so I have dropped this part for now.
> Andrew, as this is a bit tied to the buildfarm code and any
> simplifications that could happen there, do you have any comments
> and/or suggestions for this patch?



I haven't looked at the patch closely yet, but from a buildfarm POV I
think the only thing that needs to be done is to inhibit the buildfarm
client module if the TAP tests are present. The buildfarm code that runs
TAP tests should automatically detect and run the new test.

I've just counted and there are 116 animals reporting check-pg_upgrade,
so we'd better put that out pronto. It's a little early but I'll try to
push out a release containing code for it on Monday or Tuesday (it's a
one line addition).


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:
> I haven't looked at the patch closely yet, but from a buildfarm POV I
> think the only thing that needs to be done is to inhibit the buildfarm
> client module if the TAP tests are present. The buildfarm code that runs
> TAP tests should automatically detect and run the new test.

> I've just counted and there are 116 animals reporting check-pg_upgrade,
> so we'd better put that out pronto. It's a little early but I'll try to
> push out a release containing code for it on Monday or Tuesday (it's a
> one line addition).

IIUC, the only problem for a non-updated animal would be that it'd
run the test twice?  Or would it actually fail?  If the latter,
we'd need to sit on the patch rather longer.

            regards, tom lane

On 10/2/21 5:03 PM, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> I haven't looked at the patch closely yet, but from a buildfarm POV I
>> think the only thing that needs to be done is to inhibit the buildfarm
>> client module if the TAP tests are present. The buildfarm code that runs
>> TAP tests should automatically detect and run the new test.
>> I've just counted and there are 116 animals reporting check-pg_upgrade,
>> so we'd better put that out pronto. It's a little early but I'll try to
>> push out a release containing code for it on Monday or Tuesday (it's a
>> one line addition).
> IIUC, the only problem for a non-updated animal would be that it'd
> run the test twice?  Or would it actually fail?  If the latter,
> we'd need to sit on the patch rather longer.
>
>             


The patch removes test.sh, so yes it would break.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:
> On 10/2/21 5:03 PM, Tom Lane wrote:
>> IIUC, the only problem for a non-updated animal would be that it'd
>> run the test twice?  Or would it actually fail?  If the latter,
>> we'd need to sit on the patch rather longer.

> The patch removes test.sh, so yes it would break.

Maybe we could leave test.sh in place for awhile?  I'd rather
not cause a flag day for buildfarm owners.  (Also, how do we
see this working in the back branches?)

            regards, tom lane

On Sat, Oct 02, 2021 at 11:34:38PM -0400, Tom Lane wrote:
> Maybe we could leave test.sh in place for awhile?  I'd rather
> not cause a flag day for buildfarm owners.  (Also, how do we
> see this working in the back branches?)

I would be fine with test.sh staying around for now.

If we do that, though, I think that we had better remove the support
for upgrades across different major versions in test.sh, and keep this
capability in the new script.  I am not sure that a lot of people use
that to begin with, but it would be weird to support that with a
different configuration layer for both at the same time (test.sh uses
a combination of bin/ and lib/ paths, while TAP uses just installation
path to accomodate with what PostgresNode.pm is able to do).  The
patch of this thread also adds support for the load of an old dump
instead of an installcheck run of the old instance, which is something
the buildfarm could use.

I also looked two days ago at a proposal to move all the
pg_upgrade-specific SQLs into a new, separate, file that makes use of
psql's \if to do the job encoded now in test.sh.  I think that it
would be strange to duplicate this logic in a the pg_upgrade TAP test
and test.sh if we finish by keeping both around for now.  So that's a
second item we had better deal with first, in my opinion:
https://www.postgresql.org/message-id/YVa/se5gxr1PsXDy@paquier.xyz

Thoughts?
--
Michael

On 10/2/21 11:34 PM, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> On 10/2/21 5:03 PM, Tom Lane wrote:
>>> IIUC, the only problem for a non-updated animal would be that it'd
>>> run the test twice?  Or would it actually fail?  If the latter,
>>> we'd need to sit on the patch rather longer.
>> The patch removes test.sh, so yes it would break.
> Maybe we could leave test.sh in place for awhile?  I'd rather
> not cause a flag day for buildfarm owners.  (Also, how do we
> see this working in the back branches?)
>
>             


Actually, I was wrong. The module just does "make check" for non-MSVC.
For MSVC it calls vcregress.pl, which the patch doesn't touch (it
should, I think).


cheers


andrew



--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 03.10.21 09:03, Michael Paquier wrote:
> On Sat, Oct 02, 2021 at 11:34:38PM -0400, Tom Lane wrote:
>> Maybe we could leave test.sh in place for awhile?  I'd rather
>> not cause a flag day for buildfarm owners.  (Also, how do we
>> see this working in the back branches?)
> 
> I would be fine with test.sh staying around for now.

test.sh could be changed to invoke the TAP test.

On Sun, Oct 10, 2021 at 04:07:43PM +0200, Peter Eisentraut wrote:
> On 03.10.21 09:03, Michael Paquier wrote:
>> On Sat, Oct 02, 2021 at 11:34:38PM -0400, Tom Lane wrote:
>>> Maybe we could leave test.sh in place for awhile?  I'd rather
>>> not cause a flag day for buildfarm owners.  (Also, how do we
>>> see this working in the back branches?)
>>
>> I would be fine with test.sh staying around for now.
>
> test.sh could be changed to invoke the TAP test.

That would remove the possibility to run the tests of pg_upgrade with
--enable-tap-tests, which is the point I think Tom was making, because
TestUpgrade.pm in the buildfarm code just uses "make check" as of the
following:
$cmd = "cd $self->{pgsql}/src/bin/pg_upgrade && $make $instflags check";
--
Michael

On 10/10/21 10:07 AM, Peter Eisentraut wrote:
> On 03.10.21 09:03, Michael Paquier wrote:
>> On Sat, Oct 02, 2021 at 11:34:38PM -0400, Tom Lane wrote:
>>> Maybe we could leave test.sh in place for awhile?  I'd rather
>>> not cause a flag day for buildfarm owners.  (Also, how do we
>>> see this working in the back branches?)
>>
>> I would be fine with test.sh staying around for now.
>
> test.sh could be changed to invoke the TAP test.


Keeping test.sh is not necessary - I mis-remembered what the test module
does.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Sun, Oct 03, 2021 at 08:22:57AM -0400, Andrew Dunstan wrote:
> Actually, I was wrong. The module just does "make check" for non-MSVC.
> For MSVC it calls vcregress.pl, which the patch doesn't touch (it
> should, I think).

Yes, it should.  And I'd like to do things so as we replace all the
internals of upgradecheck() by a call to tap_check().  The patch does
not work yet properly with MSVC, and there were some problems in
getting the invocation of pg_regress right as far as I recall.  That's
why I have left this part for now.  I don't see why we could not do
the MSVC part as an independent step though, getting rid of test.sh is
appealing enough in itself.
--
Michael

On Mon, Oct 11, 2021 at 09:04:47AM -0400, Andrew Dunstan wrote:
> Keeping test.sh is not necessary - I mis-remembered what the test module
> does.

So..  Are people fine to remove entirely test.sh at the end, requiring
the tests of pg_upgrade to have TAP installed?  I'd rather raise the
bar here, as it would keep the code simpler in the tree in the long
term.  Or am I misunderstanding something?
--
Michael

Hi,

On 2021-10-02 23:34:38 -0400, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > On 10/2/21 5:03 PM, Tom Lane wrote:
> >> IIUC, the only problem for a non-updated animal would be that it'd
> >> run the test twice?  Or would it actually fail?  If the latter,
> >> we'd need to sit on the patch rather longer.
> 
> > The patch removes test.sh, so yes it would break.
> 
> Maybe we could leave test.sh in place for awhile?  I'd rather
> not cause a flag day for buildfarm owners.  (Also, how do we
> see this working in the back branches?)

Seems like we might get away with making make -C contrib/pg_upgrade check and
vcregress.pl upgradecheck do nothing?

For the common case of not testing cross-version stuff, pg_upgrade's tests
would just be invoked via run_build.pl:run_bin_tests(). And TestUpgrade.pm
should be fine with a test doing nothing.

We'd not loose coverage with non-updated BF animals unless they have tap tests
disabled. Just the cross-version test would need timely work by buildfarm
operators - but I think Andrew could deal with that.

Greetings,

Andres Freund

On Mon, Dec 13, 2021 at 06:08:24PM -0800, Andres Freund wrote:
> Seems like we might get away with making make -C contrib/pg_upgrade check and
> vcregress.pl upgradecheck do nothing?

You mean #contrib/#src/bin/# here, right?  I don't think that we have
any need to have "make -C" do nothing.  For vcregress.pl, we should
IMO just remove upgradecheck.

> For the common case of not testing cross-version stuff, pg_upgrade's tests
> would just be invoked via run_build.pl:run_bin_tests(). And TestUpgrade.pm
> should be fine with a test doing nothing.

Perhaps.  I am not sure what's the best picture here, TBH.  One
difference between the core stuff and the buldfarm is that in the case
of the buildfarm, we upgrade from a version that has not only the main
regression database, but everything from the contrib/ modules.

Speaking of which, I am going to send a patch for the buildfarm to be
able to use the SQL file from 0df9641, so as committers gain a bit
more control on the cross-version upgrade tests run by the buildfarm,
using the in-core code a maximum.
--
Michael

On 2021-12-14 14:31:24 +0900, Michael Paquier wrote:
> On Mon, Dec 13, 2021 at 06:08:24PM -0800, Andres Freund wrote:
> > Seems like we might get away with making make -C contrib/pg_upgrade check and
> > vcregress.pl upgradecheck do nothing?
> 
> You mean #contrib/#src/bin/# here, right?  I don't think that we have
> any need to have "make -C" do nothing.  For vcregress.pl, we should
> IMO just remove upgradecheck.

Tom's point was that the buildfarm scripts do
    if ($self->{bfconf}->{using_msvc})
        @checklog = run_log("perl vcregress.pl upgradecheck");
    else
              "cd $self->{pgsql}/src/bin/pg_upgrade && $make $instflags check";

if we don't want to break every buildfarm member that has TestUpgrade enabled
the moment this is committed, we need to have a backward compat path.

Greetings,

Andres Freund

On Mon, Dec 13, 2021 at 10:14:49PM -0800, Andres Freund wrote:
> Tom's point was that the buildfarm scripts do
>     if ($self->{bfconf}->{using_msvc})
>         @checklog = run_log("perl vcregress.pl upgradecheck");
>     else
>               "cd $self->{pgsql}/src/bin/pg_upgrade && $make $instflags check";
>
> if we don't want to break every buildfarm member that has TestUpgrade enabled
> the moment this is committed, we need to have a backward compat path.

Missed that, thanks!  I'll think about all that a bit more before
sending a long-overdue rebased version.
--
Michael

On Wed, Dec 15, 2021 at 10:47:24AM +0900, Michael Paquier wrote:
> Missed that, thanks!  I'll think about all that a bit more before
> sending a long-overdue rebased version.

Okay, here is finally a rebase of this patch, where I have fixed a
couple of existing issues, and I have extended the patch to the point
where the support range is what I expect should be.  In short:
- Added support for MSVC for the TAP test.  I have considered making
upgradecheck silent, but after thinking about it I have just filtered
pg_upgrade from bincheck, and simplified upgradecheck to launch the
new test.  It is simple to switch from one approach to another.  This
shaves some code in vcregress.pl.
- Fixed a set of issues with various chdir commands in the previous
patches.  The command of pg_regress has been tweaked so as all results
are part of src/bin/pg_upgrade/.  Any logs generated by pg_upgrade
stay in this location, same way as HEAD.
- Adapted to the new modules of src/test/perl/.
- Support for cross-upgrades now uses upgrade_adapt.sql (I have sent a
patch for the buildfarm client about that yesterday actually), same
way as test.sh on HEAD.  Like HEAD, attempting to use the
cross-version HEAD causes diffs between the old and the new dumps.
But there is nothing new here.  This could be improved more but the
attached does already a lot.
- Like the previous versions, this supports two modes when setting up
the to-be-upgraded cluster: setup things from an old dump or use
pg_regress.  The buildfarm does the former for upgrades down to 9.2.
The core code does the latter.

I may have missed one thing or two, but I think that's pretty much
what we should be looking for to do the switch to TAP in terms of
coverage.
--
Michael

On Thu, Dec 16, 2021 at 11:51:55AM +0900, Michael Paquier wrote:
> I may have missed one thing or two, but I think that's pretty much
> what we should be looking for to do the switch to TAP in terms of
> coverage.

Rebased patch to cool down the CF bot, as per the addition of
--no-sync to pg_upgrade.
--
Michael

On Wed, Jan 05, 2022 at 05:12:41PM +0900, Michael Paquier wrote:
> Rebased patch to cool down the CF bot, as per the addition of
> --no-sync to pg_upgrade.

The CF bot is unhappy, so here is a rebase, with some typo fixes
reported by Justin offlist.
--
Michael

Hi,

On Tue, Jan 11, 2022 at 04:14:25PM +0900, Michael Paquier wrote:
> The CF bot is unhappy, so here is a rebase, with some typo fixes
> reported by Justin offlist.

The cfbot still complains about this patch on Windows:
https://cirrus-ci.com/task/6411385683836928

https://api.cirrus-ci.com/v1/artifact/task/6411385683836928/tap/src/bin/pg_upgrade/tmp_check/log/regress_log_002_pg_upgrade

# Running: pg_upgrade --no-sync -d c:/cirrus/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -D
c:/cirrus/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_new_node_data/pgdata-b C:/cirrus/tmp_install/bin -B
C:/cirrus/tmp_install/bin-p 56296 -P 56297
 

libpq environment variable PGHOST has a non-local server value:
C:/Users/ContainerAdministrator/AppData/Local/Temp/FhBIlsw6SV
Failure, exiting
not ok 3 - run of pg_upgrade for new instance

#   Failed test 'run of pg_upgrade for new instance'
#   at t/002_pg_upgrade.pl line 255.
### Starting node "new_node"
# Running: pg_ctl -w -D c:/cirrus/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_new_node_data/pgdata -l
c:/cirrus/src/bin/pg_upgrade/tmp_check/log/002_pg_upgrade_new_node.log-o --cluster-name=new_node start
 
waiting for server to start.... done
server started
# Postmaster PID for node "new_node" is 5748
# Running: pg_dumpall --no-sync -d port=56297 host=C:/Users/ContainerAdministrator/AppData/Local/Temp/FhBIlsw6SV
dbname='postgres'-f C:\cirrus\src\bin\pg_upgrade\tmp_check\tmp_test_X4aZ/dump2.sql
 
# Running: diff -q C:\cirrus\src\bin\pg_upgrade\tmp_check\tmp_test_X4aZ/dump1.sql
C:\cirrus\src\bin\pg_upgrade\tmp_check\tmp_test_X4aZ/dump2.sql
Files C:\cirrus\src\bin\pg_upgrade\tmp_check\tmp_test_X4aZ/dump1.sql and
C:\cirrus\src\bin\pg_upgrade\tmp_check\tmp_test_X4aZ/dump2.sqldiffer
 
not ok 4 - old and new dump match after pg_upgrade

#   Failed test 'old and new dump match after pg_upgrade'

On Sat, Jan 15, 2022 at 01:52:39PM +0800, Julien Rouhaud wrote:
> libpq environment variable PGHOST has a non-local server value:
C:/Users/ContainerAdministrator/AppData/Local/Temp/FhBIlsw6SV
> Failure, exiting
> not ok 3 - run of pg_upgrade for new instance

There are two things here, as far as I understand:
1) This is a valid Windows path.  So shouldn't we fix pg_upgrade's
server.c to be a bit more compliant with Windows paths?  The code
accepts only paths beginning with '/' as local paths, so this breaks.
2) It looks safer in the long run to disable completely PGHOST and
PGHOSTADDR when running the pg_upgrade command in the test, and we'd
better not use Cluster::command_ok() or we would fall down to each
node's local environment.  This could be done in the tests as of the
attached, I guess, and this would bypass the problem coming from 1).

The patch needed a refresh as --make-testtablespace-dir has been
removed as of d6d317d.
--
Michael

Hi,

On 2022-01-18 11:20:16 +0900, Michael Paquier wrote:
> On Sat, Jan 15, 2022 at 01:52:39PM +0800, Julien Rouhaud wrote:
> > libpq environment variable PGHOST has a non-local server value:
C:/Users/ContainerAdministrator/AppData/Local/Temp/FhBIlsw6SV
> > Failure, exiting
> > not ok 3 - run of pg_upgrade for new instance
> 
> There are two things here, as far as I understand:
> 1) This is a valid Windows path.  So shouldn't we fix pg_upgrade's
> server.c to be a bit more compliant with Windows paths?  The code
> accepts only paths beginning with '/' as local paths, so this breaks.

It also doesn't handle @ correctly. Makes sense to fix. Should probably use
the same logic that libpq, psql, ... use?

            if (is_unixsock_path(ch->host))
                ch->type = CHT_UNIX_SOCKET;

that'd basically be the same amount of code. And easier to understand.

Greetings,

Andres Freund

Hi,

On 2022-01-18 11:20:16 +0900, Michael Paquier wrote:
> +# required for 002_pg_upgrade.pl
> +REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
> +export REGRESS_SHLIB

It seems weird to propagate this into multiple places. Why don't we define
that centrally?

Although it's weird for this to use REGRESS_SHLIB, given it's just doing
dirname() on it. 027_stream_regress.pl has the "defense" of not wanting to
duplicate the variable with 017_shm.pl...

Not that I understand why 017_shm.pl and all the regression test source
fileseven need $(DLSUFFIX) - expand_dynamic_library_name() should take care of
it?


> +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/bin/pg_upgrade
> +export REGRESS_OUTPUTDIR

I don't really understand why 027_stream_regress.pl is using this (and thus
not why it's used here). The tap tests create files all the time, why is this
different?

It's not like make / msvc put the data in different places:
src/test/recovery/Makefile:REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery/tmp_check
src/tools/msvc/vcregress.pl: $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery/tmp_check";

> +
> +# From now on, the test of pg_upgrade consists in setting up an instance.

What does "from now on" mean?



> +# Default is the location of this source code for both nodes used with
> +# the upgrade.

Can't quite parse.



> +# Initialize a new node for the upgrade.  This is done early so as it is
> +# possible to know with which node's PATH the initial dump needs to be
> +# taken.
> +my $newnode = PostgreSQL::Test::Cluster->new('new_node');
> +$newnode->init(extra => [ '--locale=C', '--encoding=LATIN1' ]);
> +my $newbindir = $newnode->config_data('--bindir');
> +my $oldbindir = $oldnode->config_data('--bindir');

Why C/LATIN?

Right now pg_upgrade test.sh uses --wal-segsize 1, and that has helped
identify several bugs. So I'd rather not give it up, even if it's a bit weird.



> +    my @regress_command = [
> +        $ENV{PG_REGRESS},
> +        '--schedule',     "$oldsrc/src/test/regress/parallel_schedule",
> +        '--bindir',       $oldnode->config_data('--bindir'),
> +        '--dlpath',       $dlpath,
> +        '--port',         $oldnode->port,
> +        '--outputdir',    $outputdir,
> +        '--inputdir',     $inputdir,
> +        '--use-existing'
> +    ];

I think this should use --host (c.f. 7340aceed72). Or is it intending to use
the host via env? If so, why is the port specified?


> +    @regress_command = (@regress_command, @extra_opts);
> +
> +    $oldnode->command_ok(@regress_command,
> +        'regression test run on old instance');

I also think this should take EXTRA_REGRESS_OPTS into account - test.sh did.


> +# After dumping, update references to the old source tree's regress.so
> +# to point to the new tree.
> +if (defined($ENV{oldinstall}))
> +{


Kinda asking for its own function...

> +
> +# Update the instance.
> +$oldnode->stop;
> +
> +# Time for the real run.

As opposed to the unreal one?


> +# pg_upgrade would complain if PGHOST, so as there are no attempts to
> +# connect to a different server than the upgraded ones.

"complain if PGHOST"?


> +# Take a second dump on the upgraded instance.

Sounds like you're taking to post-upgrade pg_dumps.




Greetings,

Andres Freund

On Sun, Feb 13, 2022 at 5:50 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-01-18 11:20:16 +0900, Michael Paquier wrote:
> > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/bin/pg_upgrade
> > +export REGRESS_OUTPUTDIR
>
> I don't really understand why 027_stream_regress.pl is using this (and thus
> not why it's used here). The tap tests create files all the time, why is this
> different?
>
> It's not like make / msvc put the data in different places:
> src/test/recovery/Makefile:REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery/tmp_check
> src/tools/msvc/vcregress.pl: $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery/tmp_check";

As I wrote in
https://www.postgresql.org/message-id/CA%2BhUKGK-%2Bmg6RWiDu0JudF6jWeL5%2BgPmi8EKUm1eAzmdbwiE_A%40mail.gmail.com,

>> > +# required for 027_stream_regress.pl
>> > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
>> > +export REGRESS_OUTPUTDIR
>>
>> Why do we need this?
>
> The Make macro "prove_check" (src/Makefile.global.in) always changes
> to the source directory to run TAP tests. Without an explicit
> directive to control where regression test output goes, it got
> splattered all over the source tree in VPATH builds. I didn't see an
> existing way to adjust that (did I miss something?). Hence desire to
> pass down a path in the build tree. Better ideas welcome.

I thought it was a goal that VPATH builds shouldn't pollute the source
tree, but the Make macro prove_check is explicitly doing so by
default.  Perhaps *that* should be fixed?

Thomas Munro <thomas.munro@gmail.com> writes:
> I thought it was a goal that VPATH builds shouldn't pollute the source
> tree, but the Make macro prove_check is explicitly doing so by
> default.  Perhaps *that* should be fixed?

Indeed.  That seems broken by definition.

More generally, I thought we'd established a convention that
all files made by TAP tests should be put inside the tmp_check
directory, to simplify cleanup and .gitignore rules.  But in
a VPATH build, tmp_check ought to live in the build tree.

            regards, tom lane

Hi,

On 2022-02-13 18:07:30 +1300, Thomas Munro wrote:
> On Sun, Feb 13, 2022 at 5:50 PM Andres Freund <andres@anarazel.de> wrote:
> >> > +# required for 027_stream_regress.pl
> >> > +REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery
> >> > +export REGRESS_OUTPUTDIR
> >>
> >> Why do we need this?
> >
> > The Make macro "prove_check" (src/Makefile.global.in) always changes
> > to the source directory to run TAP tests. Without an explicit
> > directive to control where regression test output goes, it got
> > splattered all over the source tree in VPATH builds. I didn't see an
> > existing way to adjust that (did I miss something?). Hence desire to
> > pass down a path in the build tree. Better ideas welcome.
> 
> I thought it was a goal that VPATH builds shouldn't pollute the source
> tree, but the Make macro prove_check is explicitly doing so by
> default.  Perhaps *that* should be fixed?

Sure, prove changes into the source dir. But we don't put test data / output
into the source? That's what TESTDIR is used for:

    # Determine output directories, and create them.  The base path is the
    # TESTDIR environment variable, which is normally set by the invoking
    # Makefile.
    $tmp_check = $ENV{TESTDIR} ? "$ENV{TESTDIR}/tmp_check" : "tmp_check";
    $log_path = "$tmp_check/log";

Afaics all the "regress test inside tap test" cases would need to do is to pass
--outputdir=${PostgreSQL::Test::Utils::tmp_check} and you'd get exactly the same path as
REGRESS_OUTPUTDIR currently provides.

I only use vpath builds, and I don't see any tap test data / log in the source
tree....

Greetings,

Andres Freund

On Sun, Feb 13, 2022 at 6:29 PM Andres Freund <andres@anarazel.de> wrote:
> Afaics all the "regress test inside tap test" cases would need to do is to pass
> --outputdir=${PostgreSQL::Test::Utils::tmp_check} and you'd get exactly the same path as
> REGRESS_OUTPUTDIR currently provides.

Ahh, right.  I assume it still needs perl2host() treatment for MSYS2
systems, because jacana's log shows TESTDIR is set to a Unixoid path
that I assume pg_regress's runtime can't use.  That leads to the
attached.

On 2022-02-14 11:23:18 +1300, Thomas Munro wrote:
> On Sun, Feb 13, 2022 at 6:29 PM Andres Freund <andres@anarazel.de> wrote:
> > Afaics all the "regress test inside tap test" cases would need to do is to pass
> > --outputdir=${PostgreSQL::Test::Utils::tmp_check} and you'd get exactly the same path as
> > REGRESS_OUTPUTDIR currently provides.
> 
> Ahh, right.  I assume it still needs perl2host() treatment for MSYS2
> systems, because jacana's log shows TESTDIR is set to a Unixoid path
> that I assume pg_regress's runtime can't use.  That leads to the
> attached.

Looks sane to me.

On Sun, Feb 13, 2022 at 02:55:26PM -0800, Andres Freund wrote:
> On 2022-02-14 11:23:18 +1300, Thomas Munro wrote:
>> Ahh, right.  I assume it still needs perl2host() treatment for MSYS2
>> systems, because jacana's log shows TESTDIR is set to a Unixoid path
>> that I assume pg_regress's runtime can't use.  That leads to the
>> attached.
>
> Looks sane to me.

This looks like a nice cleanup, indeed.  Nice catch.
--
Michael

On Sat, Feb 12, 2022 at 08:12:42PM -0800, Andres Freund wrote:
> It also doesn't handle @ correctly. Makes sense to fix. Should probably use
> the same logic that libpq, psql, ... use?
>
>             if (is_unixsock_path(ch->host))
>                 ch->type = CHT_UNIX_SOCKET;
>
> that'd basically be the same amount of code. And easier to understand.

So, I am catching up with some parts of this thread, and I have
managed to miss is_unixsock_path().  Except if I am missing something
(now it is close to the end of the day here), a minimal change would
be something like that as we'd still want to allow the use of
localhost and others:
    if (value && strlen(value) > 0 &&
    /* check for 'local' host values */
        (strcmp(value, "localhost") != 0 && strcmp(value, "127.0.0.1") != 0 &&
-        strcmp(value, "::1") != 0 && value[0] != '/'))
+        strcmp(value, "::1") != 0 && !is_unixsock_path(value)))

Or perhaps we should restrict more the use of localhost values for
non-WIN32?  Opinions?
--
Michael

On Sat, Feb 12, 2022 at 08:50:41PM -0800, Andres Freund wrote:
> On 2022-01-18 11:20:16 +0900, Michael Paquier wrote:
>> +# required for 002_pg_upgrade.pl
>> +REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
>> +export REGRESS_SHLIB
>
> It seems weird to propagate this into multiple places. Why don't we define
> that centrally?
>
> Although it's weird for this to use REGRESS_SHLIB, given it's just doing
> dirname() on it. 027_stream_regress.pl has the "defense" of not wanting to
> duplicate the variable with 017_shm.pl...
>
> Not that I understand why 017_shm.pl and all the regression test source
> fileseven need $(DLSUFFIX) - expand_dynamic_library_name() should take care of
> it?

I agree that we should be able to get rid of that in the long-term,
but this also feels like a separate issue to me and the patch is
already doing a lot.  I am wondering about the interactions of
installcheck with abs_top_builddir, though.  Should it be addressed
first?  It does not feel like a mandatory requirement for this
thread, anyway.

> It's not like make / msvc put the data in different places:
> src/test/recovery/Makefile:REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/test/recovery/tmp_check
> src/tools/msvc/vcregress.pl: $ENV{REGRESS_OUTPUTDIR} = "$topdir/src/test/recovery/tmp_check";

Yeah, removed.

>> +# From now on, the test of pg_upgrade consists in setting up an instance.
>
> What does "from now on" mean?

In this context, the next steps of the test.  Removed.

>> +# Default is the location of this source code for both nodes used with
>> +# the upgrade.
>
> Can't quite parse.

Reworded, to something hopefully better.

>> +# Initialize a new node for the upgrade.  This is done early so as it is
>> +# possible to know with which node's PATH the initial dump needs to be
>> +# taken.
>> +my $newnode = PostgreSQL::Test::Cluster->new('new_node');
>> +$newnode->init(extra => [ '--locale=C', '--encoding=LATIN1' ]);
>> +my $newbindir = $newnode->config_data('--bindir');
>> +my $oldbindir = $oldnode->config_data('--bindir');
>
> Why C/LATIN?

Well, these are bits from the patch that I have played with
extensively, and it took me some time to remember why this was needed.
The reason why I introduced this option is that the patch created the
database "regression" using a createdb command that would feed from
template1 as pg_regress used --use-existing.  And this combination
required to enforce --locale=C to avoid two regression diffs in
int8.sql and numeric.sql.  It is possible to simplify things by
removing --use-existing and the database creation, so as pg_regress
handles the creation of the database "regression" with template0 to
avoid any problems related to locales.

Now, if you do *not* do that, I have noticed that we run into problems
when testing the TAP script with older versions, where pg_regress
would may not create the "regression" database, hence requiring an
extra createdb (perhaps that's better with --locale=C and
--template=template0) with --use-existing present for the pg_regress
command, command coming from the old branch.

Hmm.  At the end of the day, I am wondering whether we should not give
up entirely on the concept of running the regression tests on older
branches in the TAP script of a newer branch.  pg_regress needs to
come from the old source tree, meaning that we would most likely need
to maintain a set of compatibility tweaks that would most probably
rot over the time, and the buildfarm only cares about the possibility
to set up old instances by loading dumps rather than running
pg_regress.  This would also make the switch to TAP much easier (no
need for the extra createdb or --locale AFAIK).  So attempting to
maintain all that is going to be a PITA in the long term, and there is
nothing running that automatically anyway.

There is also the extra requirement to adjust dump files, but that's
independent of setting up the old instance to upgrade, and I don't
really plan to tackle that as of this thread (note that the buildfarm
client has extra tweaks regarding that).

Any thoughts about that?

> Right now pg_upgrade test.sh uses --wal-segsize 1, and that has helped
> identify several bugs. So I'd rather not give it up, even if it's a bit weird.

--allow-group-access was missing as well.

>> +    my @regress_command = [
>> +        $ENV{PG_REGRESS},
>> +        '--schedule',     "$oldsrc/src/test/regress/parallel_schedule",
>> +        '--bindir',       $oldnode->config_data('--bindir'),
>> +        '--dlpath',       $dlpath,
>> +        '--port',         $oldnode->port,
>> +        '--outputdir',    $outputdir,
>> +        '--inputdir',     $inputdir,
>> +        '--use-existing'
>> +    ];
>
> I think this should use --host (c.f. 7340aceed72). Or is it intending to use
> the host via env? If so, why is the port specified?

Hm.  It looks like you are right here, so added.

>> +    @regress_command = (@regress_command, @extra_opts);
>> +
>> +    $oldnode->command_ok(@regress_command,
>> +        'regression test run on old instance');
>
> I also think this should take EXTRA_REGRESS_OPTS into account - test.sh did.

This is already taken into account, as of the @extra_opts bits.

>> +# After dumping, update references to the old source tree's regress.so
>> +# to point to the new tree.
>> +if (defined($ENV{oldinstall}))
>> +{
>
> Kinda asking for its own function...

I am not sure this is a gain in readability just for this part, FWIW,
and once you drop support for setting up an old instance with
pg_regress, that would not be needed.

>> +# Update the instance.
>> +$oldnode->stop;
>> +
>> +# Time for the real run.
>
> As opposed to the unreal one?

Removed that.

>> +# pg_upgrade would complain if PGHOST, so as there are no attempts to
>> +# connect to a different server than the upgraded ones.
>
> "complain if PGHOST"?

There is no need for this tweak once check_pghost_envvar() is fixed to
be able to understand Windows paths.  This was not working under the
CI on Windows anyway, but the check_pghost_envvar() fix does.

A last thing that was missing from the patch, AFAIK, is to scan the
contents of pg_upgrade_output.d/log, if anything is left around after
a failure so as the buildfarm is able to report all the logs.
pg_upgrade's .gitignore has no need for a refresh, as well.

I have split the patch set into two parts:
- 0001 is a fix for check_pghost_envvar() with the addition of a call
to is_unixsock_path() to make sure that Windows paths are handled.
This has proved to be enough to make the CI report green on Windows.
- 0002 is the test, with all the fixes and adjustments mentioned
upthread, including making sure that the tests can be run with older
branches, for now.
--
Michael

On Tue, Feb 15, 2022 at 01:02:41PM +0900, Michael Paquier wrote:
> Hmm.  At the end of the day, I am wondering whether we should not give
> up entirely on the concept of running the regression tests on older
> branches in the TAP script of a newer branch.  pg_regress needs to
> come from the old source tree, meaning that we would most likely need
> to maintain a set of compatibility tweaks that would most probably
> rot over the time, and the buildfarm only cares about the possibility
> to set up old instances by loading dumps rather than running
> pg_regress.  This would also make the switch to TAP much easier (no
> need for the extra createdb or --locale AFAIK).  So attempting to
> maintain all that is going to be a PITA in the long term, and there is
> nothing running that automatically anyway.
>
> There is also the extra requirement to adjust dump files, but that's
> independent of setting up the old instance to upgrade, and I don't
> really plan to tackle that as of this thread (note that the buildfarm
> client has extra tweaks regarding that).
>
> Any thoughts about that?

I have been looking at how much simplicity this brings, and I have to
admit that it is tempting to just support the loading of dumps when
setting up the old instance to upgrade from.  We'd still need to do an
extra effort in terms of cleaning up the diffs for the dump of the old
instance with older versions once/if this is plugged into the
buildfarm, but that could be addressed later depending on the versions
that need to be covered.
--
Michael

On Wed, Feb 16, 2022 at 01:58:10PM +0900, Michael Paquier wrote:
> I have been looking at how much simplicity this brings, and I have to
> admit that it is tempting to just support the loading of dumps when
> setting up the old instance to upgrade from.  We'd still need to do an
> extra effort in terms of cleaning up the diffs for the dump of the old
> instance with older versions once/if this is plugged into the
> buildfarm, but that could be addressed later depending on the versions
> that need to be covered.

The bug related to the detection of Windows and temporary paths for
pg_upgrade's server.c has been fixed as of dc57366, so attached is the
remaining rebased piece as perl2host has been recently removed.

Do others have an opinion about a backpatch of the bugfix?  Nobody has
complained about that since pg_upgrade exists, so I have just done the
change on HEAD.
--
Michael

On 2022-02-15 13:02:41 +0900, Michael Paquier wrote:
> >> +    @regress_command = (@regress_command, @extra_opts);
> >> +
> >> +    $oldnode->command_ok(@regress_command,
> >> +        'regression test run on old instance');
> > 
> > I also think this should take EXTRA_REGRESS_OPTS into account - test.sh did.
> 
> This is already taken into account, as of the @extra_opts bits.

But in a bad way, because EXTRA_REGRESS_OPTS now always wins, even for stuff
we want to override. Note how test.sh explicitly specifies port, bindir etc
after the pre-existing EXTRA_REGRESS_OPTS.

Hi,

On 2022-03-02 15:57:23 +0900, Michael Paquier wrote:
> Do others have an opinion about a backpatch of the bugfix?  Nobody has
> complained about that since pg_upgrade exists, so I have just done the
> change on HEAD.

WFM.



> +++ b/src/bin/pg_upgrade/t/001_basic.pl
> @@ -0,0 +1,9 @@
> +use strict;
> +use warnings;
> +
> +use PostgreSQL::Test::Utils;
> +use Test::More tests => 8;

Outdated.

> +program_help_ok('pg_upgrade');
> +program_version_ok('pg_upgrade');
> +program_options_handling_ok('pg_upgrade');

Unrelated. But I kinda wish we'd do this in a saner manner than copying this
test into every binary. E.g. by ensuring that all tools installed in the temp
install are tested or such.


> +# The test of pg_upgrade consists in setting up an instance.  This is the
> +# source instance used for the upgrade. Then a new and fresh instance is
> +# created, and is used as the target instance for the upgrade.

This seems a bit repetitive. Lots of "instance".

> Before
> +# running an upgrade, a logical dump of the old instance is taken, and a
> +# second logical dump of the new instance is taken after the upgrade.
> +# The upgrade test passes if there are no differences in these two dumps.
> +
> +# Testing upgrades with an older instance of PostgreSQL requires setting up
> +# two environment variables, as of:
> +# - "olddump", to point to a dump file that will be used to set
> +#   up the old instance to upgrade from, the dump being restored in the
> +#   old cluster.
> +# - "oldinstall", to point to the installation path of the old
> +#   instance.
> +if (   (defined($ENV{olddump}) && !defined($ENV{oldinstall}))
> +    || (!defined($ENV{olddump}) && defined($ENV{oldinstall})))

Odd indentation. Spaces between parens?


> +$newnode->init(extra => [ '--wal-segsize', '1', '--allow-group-access' ]);

I'd copy the comments from test.sh wrt --wal-segsize,
--allow-group-access.


Greetings,

Andres Freund

On Wed, Mar 02, 2022 at 12:01:17AM -0800, Andres Freund wrote:
> But in a bad way, because EXTRA_REGRESS_OPTS now always wins, even for stuff
> we want to override. Note how test.sh explicitly specifies port, bindir etc
> after the pre-existing EXTRA_REGRESS_OPTS.

Ah, right.  Will fix.
--
Michael

On Wed, Mar 02, 2022 at 12:07:29AM -0800, Andres Freund wrote:
>> +++ b/src/bin/pg_upgrade/t/001_basic.pl
>> @@ -0,0 +1,9 @@
>> +use strict;
>> +use warnings;
>> +
>> +use PostgreSQL::Test::Utils;
>> +use Test::More tests => 8;
>
> Outdated.

Fixed.

>> +program_help_ok('pg_upgrade');
>> +program_version_ok('pg_upgrade');
>> +program_options_handling_ok('pg_upgrade');
>
> Unrelated. But I kinda wish we'd do this in a saner manner than copying this
> test into every binary. E.g. by ensuring that all tools installed in the temp
> install are tested or such.

Perhaps.  I am sticking with the existing style for now.

>> +# The test of pg_upgrade consists in setting up an instance.  This is the
>> +# source instance used for the upgrade. Then a new and fresh instance is
>> +# created, and is used as the target instance for the upgrade.
>
> This seems a bit repetitive. Lots of "instance".

Indeed.  I have reworked the whole, rather than just those three
sentences.

>> +if (   (defined($ENV{olddump}) && !defined($ENV{oldinstall}))
>> +    || (!defined($ENV{olddump}) && defined($ENV{oldinstall})))
>
> Odd indentation. Spaces between parens?

Well, perltidy tells me that this is right.

>> +$newnode->init(extra => [ '--wal-segsize', '1', '--allow-group-access' ]);
>
> I'd copy the comments from test.sh wrt --wal-segsize,
> --allow-group-access.

Done.
--
Michael

On 3/3/22 00:03, Michael Paquier wrote:
>>> +if (   (defined($ENV{olddump}) && !defined($ENV{oldinstall}))
>>> +    || (!defined($ENV{olddump}) && defined($ENV{oldinstall})))
>> Odd indentation. Spaces between parens?
> Well, perltidy tells me that this is right.
>
>

Yeah, I haven't found a way to make it stop doing that :-(


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Thu, Mar 03, 2022 at 02:03:38PM +0900, Michael Paquier wrote:
> Indeed.  I have reworked the whole, rather than just those three
> sentences.

So, any particular feelings about this patch?  This has been around
for a couple of months/years now, so it could be a good time to do the
switch now rather than wait an extra year, or even the beginning of
the next release cycle.  And the buildfarm is already able to handle
that in its code based on the last release, by skipping the upgrade
check if it finds a pg_upgrade/t/ subdirectory.
--
Michael

Michael Paquier <michael@paquier.xyz> writes:
> So, any particular feelings about this patch?  This has been around
> for a couple of months/years now, so it could be a good time to do the
> switch now rather than wait an extra year, or even the beginning of
> the next release cycle.  And the buildfarm is already able to handle
> that in its code based on the last release, by skipping the upgrade
> check if it finds a pg_upgrade/t/ subdirectory.

There's still about a third of the buildfarm running older
client releases --- I count

      2 REL_8
      2 REL_10
     13 REL_11
      6 REL_12
     16 REL_13.1
     89 REL_14

How well does this patch work with pre-14 buildfarm clients?

            regards, tom lane

I wrote:
> There's still about a third of the buildfarm running older
> client releases --- I count

>       2 REL_8
>       2 REL_10
>      13 REL_11
>       6 REL_12
>      16 REL_13.1
>      89 REL_14

Wait a minute ... actually, what's most relevant here is
the population running TAP tests, which seems to be

      2 REL_8
      4 REL_11
      1 REL_12
      7 REL_13.1
     53 REL_14

So there are still some people we'd have to nag if it doesn't
work pre-v14, but fewer than I thought --- specifically,
the owners of

butterflyfish
copperhead
eelpout
elver
halibut
kittiwake
mantid
marabou
massasauga
myna
snakefly
snapper
spurfowl
tadarida

            regards, tom lane

On 2022-03-31 01:00:14 -0400, Tom Lane wrote:
> How well does this patch work with pre-14 buildfarm clients?

Looks to me like it'll just run the test twice, once via TestUpgrade, once via
taptest. It's possible that there could be trouble somehow due to duplicated
log files or something?

On Wed, Mar 30, 2022 at 10:36:16PM -0700, Andres Freund wrote:
> On 2022-03-31 01:00:14 -0400, Tom Lane wrote:
> > How well does this patch work with pre-14 buildfarm clients?
>
> Looks to me like it'll just run the test twice, once via TestUpgrade, once via
> taptest. It's possible that there could be trouble somehow due to duplicated
> log files or something?

Hmm.  TestUpgrade.pm also uses tmp_check/, and the TAP tests would
remove this path before running.  Still, all the contents of the logs
would be printed out before moving to the next tests at the end of
check-pg_upgrade.  It does not seem like this double run is going to
be an issue on this side.
--
Michael

Michael Paquier <michael@paquier.xyz> writes:
> On Wed, Mar 30, 2022 at 10:36:16PM -0700, Andres Freund wrote:
>> On 2022-03-31 01:00:14 -0400, Tom Lane wrote:
>>> How well does this patch work with pre-14 buildfarm clients?

>> Looks to me like it'll just run the test twice, once via TestUpgrade, once via
>> taptest. It's possible that there could be trouble somehow due to duplicated
>> log files or something?

> Hmm.  TestUpgrade.pm also uses tmp_check/, and the TAP tests would
> remove this path before running.  Still, all the contents of the logs
> would be printed out before moving to the next tests at the end of
> check-pg_upgrade.  It does not seem like this double run is going to
> be an issue on this side.

Well, let's go ahead with it and see what happens.  If it's too
much of a mess we can always revert.

            regards, tom lane

On Thu, Mar 31, 2022 at 09:49:50AM -0400, Tom Lane wrote:
> Well, let's go ahead with it and see what happens.  If it's too
> much of a mess we can always revert.

Okay, done after an extra round of self-review.  I have finished by
tweaking a couple of comments, and adjusted further TESTING to explain
what needs to be done to have a dump compatible with the test.  Let's
now see what goes wrong.
--
Michael

On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote:
> On Thu, Mar 31, 2022 at 09:49:50AM -0400, Tom Lane wrote:
> > Well, let's go ahead with it and see what happens.  If it's too
> > much of a mess we can always revert.
> 
> Okay, done after an extra round of self-review.  I have finished by
> tweaking a couple of comments, and adjusted further TESTING to explain
> what needs to be done to have a dump compatible with the test.  Let's
> now see what goes wrong.

The REL_14 buildfarm client did not grab logs from the first failure:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2022-04-01%2001%3A39%3A04

The failure looked like this:

# Running: diff -q
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump1.sql
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump2.sql
/usr/bin/diff: illegal option -- q
usage: diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2
       diff [-bitw] [-C number | -U number] file1 file2
       diff [-bitw] [-D string] file1 file2
       diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s] [-S name] directory1 directory2
not ok 4 - old and new dump match after pg_upgrade

On Thu, Mar 31, 2022 at 08:42:41PM -0700, Noah Misch wrote:
> On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote:
> > On Thu, Mar 31, 2022 at 09:49:50AM -0400, Tom Lane wrote:
> > > Well, let's go ahead with it and see what happens.  If it's too
> > > much of a mess we can always revert.
> > 
> > Okay, done after an extra round of self-review.  I have finished by
> > tweaking a couple of comments, and adjusted further TESTING to explain
> > what needs to be done to have a dump compatible with the test.  Let's
> > now see what goes wrong.
> 
> The REL_14 buildfarm client did not grab logs from the first failure:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=wrasse&dt=2022-04-01%2001%3A39%3A04
> 
> The failure looked like this:
> 
> # Running: diff -q
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump1.sql
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump2.sql
> /usr/bin/diff: illegal option -- q
> usage: diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2
>        diff [-bitw] [-C number | -U number] file1 file2
>        diff [-bitw] [-D string] file1 file2
>        diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s] [-S name] directory1 directory2
> not ok 4 - old and new dump match after pg_upgrade

Is diff -q defined somewhere ?  I can't find it in postgres sources nor in
sources for bf client.

Maybe your bf member could use git diff --exit-code --quiet ?

-- 
Justin

On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote:
> Okay, done after an extra round of self-review.  I have finished by
> tweaking a couple of comments, and adjusted further TESTING to explain
> what needs to be done to have a dump compatible with the test.  Let's
> now see what goes wrong.

So, the first reports are published, and the buildfarm is rather cool
on the matter.  wrasse is the only buildfarm member that has reported
a failure, complaining that the dumps generated do not match.  I am
not completely sure what's going on there, so I have applied an extra
patch to get more information from the logs on failures, and switched
the test to use File::Compare::compare() to check if the dumps match.
This last part feels safer in the long run, anyway.  There should be a
diff command as previous runs used test.sh, so perhaps this is an
issue with its perl.  The next report should tell more.
--
Michael

On Thu, Mar 31, 2022 at 08:42:41PM -0700, Noah Misch wrote:
> The failure looked like this:
>
> # Running: diff -q
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump1.sql
/export/home/nm/farm/studio64v12_6/HEAD/pgsql.build/src/bin/pg_upgrade/tmp_check/tmp_test_lPFv/dump2.sql
> /usr/bin/diff: illegal option -- q
> usage: diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2
>        diff [-bitw] [-C number | -U number] file1 file2
>        diff [-bitw] [-D string] file1 file2
>        diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s] [-S name] directory1 directory2
> not ok 4 - old and new dump match after pg_upgrade

Ah, thanks for the information!  So the problem was that the first
commit of the patch took the diff command from the MSVC scripts, and
there is no -q on Solaris 11.3.  Using File::Compare should be enough
to fix the problem, then.  Hopefully.
--
Michael

On Thu, Mar 31, 2022 at 10:51:59PM -0500, Justin Pryzby wrote:
> Is diff -q defined somewhere ?  I can't find it in postgres sources nor in
> sources for bf client.

322becb has added such a call, at the end of 002_pg_upgrade.pl.
vcregress.pl also has one before this commit.
--
Michael

On Fri, Apr 01, 2022 at 03:01:38PM +0900, Michael Paquier wrote:
> On Thu, Mar 31, 2022 at 10:51:59PM -0500, Justin Pryzby wrote:
>> Is diff -q defined somewhere ?  I can't find it in postgres sources nor in
>> sources for bf client.
>
> 322becb has added such a call, at the end of 002_pg_upgrade.pl.
> vcregress.pl also has one before this commit.

The Windows animals seem to be in good shape, except hamerkop that
dies on "vcregress upgradecheck" when the TAP tests are disabled
causing the buildfarm client to stop.  My idea to use upgradecheck
leads to more code than just moving the test to bincheck so let's
reuse the suggestion from Andres upthread and disable completely
upgradecheck, keeping the target around only for compatibility.  The
attached does that, and the test of pg_upgrade would go through
bincheck instead.

It is late here, I'll try to get that patched up tomorrow.
--
Michael

On Fri, Apr 01, 2022 at 08:53:10PM +0900, Michael Paquier wrote:
> On Fri, Apr 01, 2022 at 03:01:38PM +0900, Michael Paquier wrote:
> > On Thu, Mar 31, 2022 at 10:51:59PM -0500, Justin Pryzby wrote:
> >> Is diff -q defined somewhere ?  I can't find it in postgres sources nor in
> >> sources for bf client.
> > 
> > 322becb has added such a call, at the end of 002_pg_upgrade.pl.
> > vcregress.pl also has one before this commit.
> 
> The Windows animals seem to be in good shape, except hamerkop that
> dies on "vcregress upgradecheck" when the TAP tests are disabled
> causing the buildfarm client to stop.  My idea to use upgradecheck
> leads to more code than just moving the test to bincheck so let's
> reuse the suggestion from Andres upthread and disable completely
> upgradecheck, keeping the target around only for compatibility.  The
> attached does that, and the test of pg_upgrade would go through
> bincheck instead.

If you do that, should also remove upgradecheck from .cirrus.yaml, which
currently runs the upgradecheck target.

I suspect this'll cause windows CI a bit slower.
https://cirrus-ci.com/task/4703731324289024

An alternative to your patch to have the buildfarm client avoid calling
upgradecheck if tap tests are disabled.  Under your patch, upgrade check is a
NOP, so it should stop calling upgradecheck anyway.  So maybe this is a better
option ?

-- 
Justin

On Fri, Apr 01, 2022 at 08:34:34AM -0500, Justin Pryzby wrote:
> If you do that, should also remove upgradecheck from .cirrus.yaml, which
> currently runs the upgradecheck target.

Indeed.  It makes no sense to keep that.  I have removed this part and
applied the patch, after one extra run through the CI.

> An alternative to your patch to have the buildfarm client avoid calling
> upgradecheck if tap tests are disabled.  Under your patch, upgrade check is a
> NOP, so it should stop calling upgradecheck anyway.  So maybe this is a better
> option ?

Yeah, there is an extra issue with the buildfarm client here.  The
animals that have TAP enabled are now running the tests of pg_upgrade
twice: once per the optional module TestUpgrade and once in
run_bin_tests()@run_build.pl.  This is something that needs to be
changed in the client code itself, and maybe the best fix is to
disable TestUpgrade.pm when running with v15~ or a newer version.  A
fix with this approach would become much easier once REL_15_STABLE is
created, though.  I am pretty sure that it should also be possible to
change the list of optional modules depending on the branch running,
but I have not dug into that..
--
Michael

On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote:
> On Thu, Mar 31, 2022 at 09:49:50AM -0400, Tom Lane wrote:
> > Well, let's go ahead with it and see what happens.  If it's too
> > much of a mess we can always revert.
> 
> Okay, done after an extra round of self-review.

commit 322becb wrote:
> --- /dev/null
> +++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl

> +# Generate a database with a name made of a range of ASCII characters.
> +sub generate_db
> +{
> +    my ($node, $from_char, $to_char) = @_;
> +
> +    my $dbname = '';
> +    for my $i ($from_char .. $to_char)
> +    {
> +        next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
> +        $dbname = $dbname . sprintf('%c', $i);
> +    }
> +    $node->run_log(
> +        [ 'createdb', '--host', $node->host, '--port', $node->port, $dbname ]
> +    );

Nothing checks the command result, so the test file passes even if each of
these createdb calls fails.  Other run_log() calls in this file have the same
problem.  This particular one should be command_ok() or similar.

--host and --port are redundant in a PostgreSQL::Test::Cluster::run_log call,
because that call puts equivalent configuration in the environment.  Other
calls in the file have the same redundant operands.  (No other test file has
redundant --host or --port.)

> +    # Grab any regression options that may be passed down by caller.
> +    my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";

Typo: s/_OPT/_OPTS/

> +    my @extra_opts     = split(/\s+/, $extra_opts_val);

src/test/recovery/t/027_stream_regress.pl and the makefiles treat
EXTRA_REGRESS_OPTS as a shell fragment.  To be compatible, use the
src/test/recovery/t/027_stream_regress.pl approach.  Affected usage patetrns
are not very important, but since the tree has code for it, you may as well
borrow that code.  These examples witness the difference:

EXTRA_REGRESS_OPTS='--nosuc" h"' MAKEFLAGS= make -C src/bin/pg_upgrade check PROVE_TESTS=t/002_pg_upgrade.pl
# log has: /home/nm/src/pg/postgresql/src/bin/pg_upgrade/../../../src/test/regress/pg_regress: unrecognized option
'--nosuc"'
EXTRA_REGRESS_OPTS='--nosuc" h"' MAKEFLAGS= make -C src/test/recovery check PROVE_TESTS=t/027_stream_regress.pl
# log has: /home/nm/src/pg/postgresql/src/test/recovery/../../../src/test/regress/pg_regress: unrecognized option
'--nosuch'
 

> --- a/src/bin/pg_upgrade/test.sh
> +++ /dev/null

> -# Create databases with names covering the ASCII bytes other than NUL, BEL,
> -# LF, or CR.  BEL would ring the terminal bell in the course of this test, and
> -# it is not otherwise a special case.  PostgreSQL doesn't support the rest.
> -dbname1=`awk 'BEGIN { for (i= 1; i < 46; i++)
> -    if (i != 7 && i != 10 && i != 13) printf "%c", i }' </dev/null`
> -# Exercise backslashes adjacent to double quotes, a Windows special case.
> -dbname1='\"\'$dbname1'\\"\\\'

This rewrite dropped the exercise of backslashes adjacent to double quotes.

On Sun, May 01, 2022 at 09:27:18PM -0700, Noah Misch wrote:
> commit 322becb wrote:

Thanks, Noah.  I am out this week, but I should be able to address all
your points at the beginning of next week.  I have added an open item
for now.
--
Michael

On Sun, May 01, 2022 at 09:27:18PM -0700, Noah Misch wrote:
> On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote:
>> commit 322becb wrote:
>
> Nothing checks the command result, so the test file passes even if each of
> these createdb calls fails.  Other run_log() calls in this file have the same
> problem.  This particular one should be command_ok() or similar.

All of them could rely on command_ok(), as they should never fail, so
switched this way.

> --host and --port are redundant in a PostgreSQL::Test::Cluster::run_log call,
> because that call puts equivalent configuration in the environment.  Other
> calls in the file have the same redundant operands.  (No other test file has
> redundant --host or --port.)

Right.  Removed all that.

>> +    # Grab any regression options that may be passed down by caller.
>> +    my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";
>
> Typo: s/_OPT/_OPTS/

Oops, fixed.

>> +    my @extra_opts     = split(/\s+/, $extra_opts_val);
>
> src/test/recovery/t/027_stream_regress.pl and the makefiles treat
> EXTRA_REGRESS_OPTS as a shell fragment.  To be compatible, use the
> src/test/recovery/t/027_stream_regress.pl approach.  Affected usage patetrns
> are not very important, but since the tree has code for it, you may as well
> borrow that code.  These examples witness the difference:

So the pattern of EXTRA_REGRESS_OPTS being used in the Makefiles is
the decision point here.  Makes sense.

>> -# Create databases with names covering the ASCII bytes other than NUL, BEL,
>> -# LF, or CR.  BEL would ring the terminal bell in the course of this test, and
>> -# it is not otherwise a special case.  PostgreSQL doesn't support the rest.
>> -dbname1=`awk 'BEGIN { for (i= 1; i < 46; i++)
>> -    if (i != 7 && i != 10 && i != 13) printf "%c", i }' </dev/null`
>> -# Exercise backslashes adjacent to double quotes, a Windows special case.
>> -dbname1='\"\'$dbname1'\\"\\\'
>
> This rewrite dropped the exercise of backslashes adjacent to double quotes.

Damn, thanks.  If I am reading that right, this could be done with the
following addition in generate_db(), adding double quotes surrounded
by backslashes before and after the database name:
$dbname = '\\"\\' . $dbname . '\\"\\';

All these fixes lead me to the attached patch.  Does that look fine to
you?

Thanks,
--
Michael

On Mon, May 09, 2022 at 12:18:39PM +0900, Michael Paquier wrote:
> All these fixes lead me to the attached patch.

I have applied this stuff as of 7dd3ee5, in time for beta1, and closed
the open item.  One difference is that I've added one backslash
surrounding the double quote at the beginning *and* the end of the
database name in the patch.  However, the original case was different,
with:
- At the beginning of the database name, one backslash before and
after the double quote.
- At the end of the database name, two backslaces before the double
quote and three after the double quote.
--
Michael

On Wed, May 11, 2022 at 10:29:44AM +0900, Michael Paquier wrote:
> On Mon, May 09, 2022 at 12:18:39PM +0900, Michael Paquier wrote:
> > All these fixes lead me to the attached patch.
> 
> I have applied this stuff as of 7dd3ee5, in time for beta1, and closed
> the open item.  One difference is that I've added one backslash
> surrounding the double quote at the beginning *and* the end of the
> database name in the patch.  However, the original case was different,
> with:
> - At the beginning of the database name, one backslash before and
> after the double quote.
> - At the end of the database name, two backslaces before the double
> quote and three after the double quote.

Why did you discontinue testing the longstanding test database name?

On Tue, May 10, 2022 at 10:32:55PM -0700, Noah Misch wrote:
> Why did you discontinue testing the longstanding test database name?

I am not sure what you mean here.  Are you saying that the test should
be changed to prefix each database name by "regression", as it was the
case in test.sh?  Or do you mean that the backslash/double-quote
business should only apply to the first database name and not the
other two, implying that the new generate_db() in 002_pg_upgrade.pl
had better have a $prefix and a $suffix like it was originally
written?
--
Michael

On Thu, May 12, 2022 at 02:27:30PM +0900, Michael Paquier wrote:
> On Tue, May 10, 2022 at 10:32:55PM -0700, Noah Misch wrote:
> > On Wed, May 11, 2022 at 10:29:44AM +0900, Michael Paquier wrote:
> > > On Mon, May 09, 2022 at 12:18:39PM +0900, Michael Paquier wrote:
> > > > All these fixes lead me to the attached patch.
> > > 
> > > I have applied this stuff as of 7dd3ee5, in time for beta1, and closed
> > > the open item.  One difference is that I've added one backslash
> > > surrounding the double quote at the beginning *and* the end of the
> > > database name in the patch.  However, the original case was different,
> > > with:
> > > - At the beginning of the database name, one backslash before and
> > > after the double quote.
> > > - At the end of the database name, two backslaces before the double
> > > quote and three after the double quote.

Here, you describe differences between test.sh and your rewrite of test.sh.

> > Why did you discontinue testing the longstanding test database name?
> 
> I am not sure what you mean here.

Here, I requested the rationale for the differences you had just described.
You made a choice to stop testing one list of database names and start testing
a different list of database names.  Why?

> Are you saying that the test should
> be changed to prefix each database name by "regression", as it was the
> case in test.sh?  Or do you mean that the backslash/double-quote
> business should only apply to the first database name and not the
> other two, implying that the new generate_db() in 002_pg_upgrade.pl
> had better have a $prefix and a $suffix like it was originally
> written?

No, I wasn't saying any of those.  (Later, I may say one or more of those.)

On Sat, May 14, 2022 at 01:27:28AM -0700, Noah Misch wrote:
> Here, I requested the rationale for the differences you had just described.
> You made a choice to stop testing one list of database names and start testing
> a different list of database names.  Why?

Because the shape of the new names does not change the test coverage
("regression" prefix or the addition of the double quotes with
backslashes for all the database names), while keeping the code a bit
simpler.  If you think that the older names are more adapted, I have
no objections to use them, FWIW, which is something like the patch
attached would achieve.

This uses the same convention as vcregress.pl before 322becb, but not
the one of test.sh where "regression" was appended to the database
names.
--
Michael

On Mon, May 16, 2022 at 02:30:00PM +0900, Michael Paquier wrote:
> On Sat, May 14, 2022 at 01:27:28AM -0700, Noah Misch wrote:
> > Here, I requested the rationale for the differences you had just described.
> > You made a choice to stop testing one list of database names and start testing
> > a different list of database names.  Why?
> 
> Because the shape of the new names does not change the test coverage
> ("regression" prefix or the addition of the double quotes with
> backslashes for all the database names), while keeping the code a bit
> simpler.  If you think that the older names are more adapted, I have
> no objections to use them, FWIW, which is something like the patch
> attached would achieve.
> 
> This uses the same convention as vcregress.pl before 322becb, but not
> the one of test.sh where "regression" was appended to the database
> names.

I would have picked the test.sh names, both because test.sh was the senior
implementation and because doing so avoids warnings under
-DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS.  See the warnings here:

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=longfin&dt=2022-05-18%2000%3A59%3A35&stg=pg_upgrade-check

More-notable line from that same log:
sh: /Users/buildfarm/bf-data/HEAD/pgsql.build/src/bin/pg_upgrade/../../../src/test/regress/pg_regress--port=5678: No
suchfile or directory
 

Commit 7dd3ee5 adopted much of the 027_stream_regress.pl approach to running
pg_regress, but it didn't grab the "is($rc, 0, 'regression tests pass')"
needed to make defects like that report a failure.

> --- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
> +++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
> @@ -13,18 +13,16 @@ use Test::More;
>  # Generate a database with a name made of a range of ASCII characters.
>  sub generate_db
>  {
> -    my ($node, $from_char, $to_char) = @_;
> +    my ($node, $prefix, $from_char, $to_char, $suffix) = @_;
>  
> -    my $dbname = '';
> +    my $dbname = $prefix;
>      for my $i ($from_char .. $to_char)
>      {
>          next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
>          $dbname = $dbname . sprintf('%c', $i);
>      }
>  
> -    # Exercise backslashes adjacent to double quotes, a Windows special
> -    # case.
> -    $dbname = '\\"\\' . $dbname . '\\\\"\\\\\\';
> +    $dbname .= $suffix;
>      $node->command_ok([ 'createdb', $dbname ]);
>  }
>  
> @@ -79,10 +77,12 @@ else
>  {
>      # Default is to use pg_regress to set up the old instance.
>  
> -    # Create databases with names covering most ASCII bytes
> -    generate_db($oldnode, 1,  45);
> -    generate_db($oldnode, 46, 90);
> -    generate_db($oldnode, 91, 127);
> +    # Create databases with names covering most ASCII bytes.  The
> +    # first name exercises backslashes adjacent to double quotes, a
> +    # Windows special case.
> +    generate_db($oldnode, "\\\"\\", 1,  45,  "\\\\\"\\\\\\");
> +    generate_db($oldnode, '',       46, 90,  '');
> +    generate_db($oldnode, '',       91, 127, '');

Does this pass on Windows?  I'm 65% confident that released IPC::Run can't
handle this input due to https://github.com/toddr/IPC-Run/issues/142.  If it's
passing for you on Windows, then disregard.

On Wed, May 18, 2022 at 01:03:15AM -0700, Noah Misch wrote:
> On Mon, May 16, 2022 at 02:30:00PM +0900, Michael Paquier wrote:
>> Because the shape of the new names does not change the test coverage
>> ("regression" prefix or the addition of the double quotes with
>> backslashes for all the database names), while keeping the code a bit
>> simpler.  If you think that the older names are more adapted, I have
>> no objections to use them, FWIW, which is something like the patch
>> attached would achieve.
>>
>> This uses the same convention as vcregress.pl before 322becb, but not
>> the one of test.sh where "regression" was appended to the database
>> names.
>
> I would have picked the test.sh names, both because test.sh was the senior
> implementation and because doing so avoids warnings under
> -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS.  See the warnings here:
>
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=longfin&dt=2022-05-18%2000%3A59%3A35&stg=pg_upgrade-check

Yes, I saw that.  This did not bother me much as the TAP tests run in
isolation, but I am fine to stick to your option and silence these.

> More-notable line from that same log:
> sh: /Users/buildfarm/bf-data/HEAD/pgsql.build/src/bin/pg_upgrade/../../../src/test/regress/pg_regress--port=5678: No
suchfile or directory 

So you are using EXTRA_REGRESS_OPTS, then, and a space is missing from
the first argument of the command used to make that work properly.

> Commit 7dd3ee5 adopted much of the 027_stream_regress.pl approach to running
> pg_regress, but it didn't grab the "is($rc, 0, 'regression tests pass')"
> needed to make defects like that report a failure.

Okay, added this one.

>> +    generate_db($oldnode, "\\\"\\", 1,  45,  "\\\\\"\\\\\\");
>> +    generate_db($oldnode, '',       46, 90,  '');
>> +    generate_db($oldnode, '',       91, 127, '');
>
> Does this pass on Windows?  I'm 65% confident that released IPC::Run can't
> handle this input due to https://github.com/toddr/IPC-Run/issues/142.  If it's
> passing for you on Windows, then disregard.

Hmm.  The CI has been passing for me with this name pattern in place,
as of https://github.com/michaelpq/postgres/tree/upgrade_tap_fixes.

Attached is an updated patch to address your concerns.
--
Michael

On Wed, May 18, 2022 at 06:20:08PM +0900, Michael Paquier wrote:
> Attached is an updated patch to address your concerns.

Looks successful.

On Fri, May 20, 2022 at 06:28:01PM -0700, Noah Misch wrote:
> Looks successful.

Thanks a lot for confirming.  I have applied that on HEAD, then.
--
Michael

Hi,

I just saw a pg_upgrade failure on my aio branch [1]. Not sure what caused it
yet. The reason I'm writing in this thread is that I looked at the
regress_log_* for the failure, and found it to be 14.95MiB (which crashed the
browser on my phone...).


https://api.cirrus-ci.com/v1/artifact/task/5167740683026432/log/src/bin/pg_upgrade/tmp_check/log/regress_log_002_pg_upgrade

That seems way beyond reasonable.

regress_log_002_pg_upgrade.log includes all of 002_pg_upgrade_old_node.log and
002_pg_upgrade_new_node.log. The old node's log includes all pg_dump queries.

Followed by many MB of diff due to

=== diff of /Users/admin/pgsql/src/bin/pg_upgrade/tmp_check/tmp_test_Q7GQ/dump1.sql and
/Users/admin/pgsql/src/bin/pg_upgrade/tmp_check/tmp_test_Q7GQ/dump2.sql
=== stdout ===

Greetings,

Andres Freund


[1] https://cirrus-ci.com/task/5167740683026432

On Fri, Jun 03, 2022 at 12:53:18PM -0700, Andres Freund wrote:
> [...]

TRAP: FailedAssertion("AmIoWorkerProcess()", File: "xlog.c", Line:
4860, PID: 35325)

> regress_log_002_pg_upgrade.log includes all of 002_pg_upgrade_old_node.log and
> 002_pg_upgrade_new_node.log. The old node's log includes all pg_dump queries.

log_statement = all is the part biting here.  It does not seem like
we'd lose a lot of context even if this is made less verbose.

> Followed by many MB of diff due to
>
> === diff of /Users/admin/pgsql/src/bin/pg_upgrade/tmp_check/tmp_test_Q7GQ/dump1.sql and
/Users/admin/pgsql/src/bin/pg_upgrade/tmp_check/tmp_test_Q7GQ/dump2.sql
> === stdout ===

Something like 80~85% of the bloat comes from the diffs in your case.
Well, it is always possible to limit that to an arbitrary amount of
characters (say 50k~100k?) to still give some context, and dump the
whole in a different file outside the log/ path (aka tmp_check/), so
that the buildfarm would show a minimum amount of information, while
local failures would still have an access to everything.

Do you have any preferences?
--
Michael

On Sat, Jun 04, 2022 at 12:35:45PM +0900, Michael Paquier wrote:
> Something like 80~85% of the bloat comes from the diffs in your case.
> Well, it is always possible to limit that to an arbitrary amount of
> characters (say 50k~100k?) to still give some context, and dump the
> whole in a different file outside the log/ path (aka tmp_check/), so
> that the buildfarm would show a minimum amount of information, while
> local failures would still have an access to everything.

After looking a bit around that.  Something like the attached, where
the characters are limited at 10k, would limit the output generated..
--
Michael