Re: Testing Extension Upgrade Paths - Mailing list pgsql-hackers

From Craig Ringer
Subject Re: Testing Extension Upgrade Paths
Date
Msg-id CAMsr+YGj8vjY5x2-qXsCcM4p839TC_T_TK1DKhJtyd-qM1U2rQ@mail.gmail.com
Whole thread Raw
In response to Re: Testing Extension Upgrade Paths  (Michael Paquier <michael.paquier@gmail.com>)
Responses Re: Testing Extension Upgrade Paths
List pgsql-hackers
On 12 December 2017 at 07:25, Michael Paquier <michael.paquier@gmail.com> wrote:
On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finzelj@gmail.com> wrote:
> I understand how to setup a regression suite for a postgres extension, but
> what I'm not clear on from the docs is if there is a pattern that exists for
> testing not only the latest version of an extension, but also an upgraded
> previous version.
>
> Is there any straightforward way to do this that doesn't involve manually
> copying tests?

It is perfectly possible to do tests on a specific version and test
upgrade paths using CREATE EXTENSION and ALTER EXTENSION which support
versioning in a SQL regression script, say:
CREATE EXTENSION foo VERSION "0.1";
-- Do stuff
ALTER EXTENSION foo UPDATE TO "0.2";
-- Do other stuff

This works well when you want to test 1.1 binaries with a 1.0 SQL extension definition.

It's not effective if you need to test 1.0 binaries+extension, then an upgrade to 1.1 binaries and extension. You have no way to install and run the 1.0 binaries without going outside pg_regress / TAP and using an external test harness/framework/script. I've been bitten by this multiple times before. For example, if 1.1 always assigns a value to some column that was nullable in 1.0, you're never going to exercise 1.1's handling of nulls in that column.

It also doesn't lend its self to complex multi-path upgrades, where for example you could upgrade 1.0.0 -> 1.0.1 -> 1.1.0 or  upgrade directly from 1.0.0 -> 1.1.0. This rapidly becomes an issue when you release 1.0.0, release 1.1.0, then release a maintenance 1.0.1  version. Now you have toadd the 1.0.1 -> 1.1.0 upgrade path, but you still have the 1.0.0 -> 1.1.0 path.

You can handle that with TAP if you're willing to write something to do the various combinations of steps and exercise them. It's not practical with pg_regress. 

More thorough testing benefits from an external harness.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

pgsql-hackers by date:

Previous
From: Amit Langote
Date:
Subject: Boolean partitions syntax
Next
From: Craig Ringer
Date:
Subject: Re: proposal: alternative psql commands quit and exit