Thread: PG_TEST_EXTRA and meson
Hi All,
Using PG_TEST_EXTRA with make is simple, one just sets that environment variable
$ make check
... snip ...
PG_REGRESS='/home/ashutosh/work/units/pghead_make/coderoot/pg/src/test/modules/xid_wraparound/../../../../src/test/regress/pg_regress'
/usr/bin/prove -I ../../../../src/test/perl/ -I . t/*.pl
# +++ tap check in src/test/modules/xid_wraparound +++
t/001_emergency_vacuum.pl .. skipped: test xid_wraparound not enabled
in PG_TEST_EXTRA
t/002_limits.pl ............ skipped: test xid_wraparound not enabled
in PG_TEST_EXTRA
t/003_wraparounds.pl ....... skipped: test xid_wraparound not enabled
in PG_TEST_EXTRA
Files=3, Tests=0, 1 wallclock secs ( 0.02 usr 0.00 sys + 0.20 cusr
0.03 csys = 0.25 CPU)
Result: NOTESTS
Set PG_TEST_EXTRA
$ PG_TEST_EXTRA=xid_wraparound make check
PG_REGRESS='/home/ashutosh/work/units/pghead_make/coderoot/pg/src/test/modules/xid_wraparound/../../../../src/test/regress/pg_regress'
/usr/bin/prove -I ../../../../src/test/perl/ -I . t/*.pl
# +++ tap check in src/test/modules/xid_wraparound +++
t/001_emergency_vacuum.pl .. ok
t/002_limits.pl ............ ok
t/003_wraparounds.pl ....... ok
All tests successful.
Files=3, Tests=11, 181 wallclock secs ( 0.03 usr 0.00 sys + 2.87
cusr 3.10 csys = 6.00 CPU)
Result: PASS
But this simple trick does not work with meson
$ meson test -C $BuildDir --suite setup --suite xid_wraparound
ninja: Entering directory `/home/ashutosh/work/units/pg_review/build_dev'
ninja: no work to do.
1/6 postgresql:setup / tmp_install
OK 0.85s
2/6 postgresql:setup / install_test_files
OK 0.06s
3/6 postgresql:setup / initdb_cache
OK 1.57s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
SKIP 0.24s
5/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
SKIP 0.26s
6/6 postgresql:xid_wraparound / xid_wraparound/002_limits
SKIP 0.27s
$ PG_TEST_EXTRA=xid_wraparound meson test -C $BuildDir --suite setup
--suite xid_wraparound
ninja: Entering directory `/home/ashutosh/work/units/pg_review/build_dev'
ninja: no work to do.
1/6 postgresql:setup / tmp_install
OK 0.41s
2/6 postgresql:setup / install_test_files
OK 0.06s
3/6 postgresql:setup / initdb_cache
OK 1.57s
4/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
SKIP 0.20s
5/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
SKIP 0.24s
6/6 postgresql:xid_wraparound / xid_wraparound/002_limits
SKIP 0.24s
the tests are still skipped.
In order to run these tests, we have to run meson setup again. There
are couple of problems with this
1. It's not clear why the tests were skipped. Also not clear that we
have to run meson setup again - from the output alone
2. Running meson setup again is costly, every time we have to run a
new test from PG_TEST_EXTRA.
3. Always configuring meson with PG_TEST_EXTRA means we will run heavy
tests every time meson test is run. I didn't find a way to not run
these tests as part of meson test once configured this way.
We should either allow make like behaviour or provide a way to not run
these tests even if configured that way.
--
Best Wishes,
Ashutosh Bapat
Hi, On Thu, 11 Jul 2024 at 09:30, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > In order to run these tests, we have to run meson setup again. There > are couple of problems with this > 1. It's not clear why the tests were skipped. Also not clear that we > have to run meson setup again - from the output alone > 2. Running meson setup again is costly, every time we have to run a > new test from PG_TEST_EXTRA. > 3. Always configuring meson with PG_TEST_EXTRA means we will run heavy > tests every time meson test is run. I didn't find a way to not run > these tests as part of meson test once configured this way. > > We should either allow make like behaviour or provide a way to not run > these tests even if configured that way. I have a two quick solutions to this: 1- More like make behaviour. PG_TEST_EXTRA is able to be set from the setup command, delete this feature so it could be set only from the environment. Then use it from the environment. 2- If PG_TEST_EXTRA is set from the setup command, use it from the setup command and discard the environment variable. If PG_TEST_EXTRA is not set from the setup command, then use it from the environment. I hope these patches help. -- Regards, Nazir Bilal Yavuz Microsoft
Attachment
On Tue, Jul 16, 2024 at 2:12 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > 2- If PG_TEST_EXTRA is set from the setup command, use it from the > setup command and discard the environment variable. If PG_TEST_EXTRA > is not set from the setup command, then use it from the environment. Is there a way for the environment to override the Meson setting rather than vice-versa? My vote would be to have both available, but with the implementation in patch 2 I'd still have to reconfigure if I wanted to change my test setup. Thanks! --Jacob
Hi, On Wed, 17 Jul 2024 at 00:27, Jacob Champion <jacob.champion@enterprisedb.com> wrote: > > On Tue, Jul 16, 2024 at 2:12 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > > > 2- If PG_TEST_EXTRA is set from the setup command, use it from the > > setup command and discard the environment variable. If PG_TEST_EXTRA > > is not set from the setup command, then use it from the environment. > > Is there a way for the environment to override the Meson setting > rather than vice-versa? My vote would be to have both available, but > with the implementation in patch 2 I'd still have to reconfigure if I > wanted to change my test setup. I think something like attached does the trick. I did not test it extensively but it passed the couple of tests I tried. -- Regards, Nazir Bilal Yavuz Microsoft
Attachment
On Wed, Jul 17, 2024 at 3:31 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Hi,
>
> On Wed, 17 Jul 2024 at 00:27, Jacob Champion
> <jacob.champion@enterprisedb.com> wrote:
> >
> > On Tue, Jul 16, 2024 at 2:12 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > >
> > > 2- If PG_TEST_EXTRA is set from the setup command, use it from the
> > > setup command and discard the environment variable. If PG_TEST_EXTRA
> > > is not set from the setup command, then use it from the environment.
> >
> > Is there a way for the environment to override the Meson setting
> > rather than vice-versa? My vote would be to have both available, but
> > with the implementation in patch 2 I'd still have to reconfigure if I
> > wanted to change my test setup.
>
> I think something like attached does the trick. I did not test it
> extensively but it passed the couple of tests I tried.
Thanks a lot for working on this.
I tested this patch with xid_wraparound. It seems to be working.
$ mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.36s
2/6 postgresql:setup / install_test_files
OK 0.10s
3/6 postgresql:setup / initdb_cache
OK 1.14s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
SKIP 0.14s
5/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
SKIP 0.14s
6/6 postgresql:xid_wraparound / xid_wraparound/002_limits
SKIP 0.14s
... snip
$ PG_TEST_EXTRA=xid_wraparound mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.38s
2/6 postgresql:setup / install_test_files
OK 0.07s
3/6 postgresql:setup / initdb_cache
OK 1.13s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
OK 67.33s 7 subtests passed
5/6 postgresql:xid_wraparound / xid_wraparound/002_limits
OK 70.14s 3 subtests passed
6/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
OK 178.01s 1 subtests passed
... snip
$ mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.36s
2/6 postgresql:setup / install_test_files
OK 0.06s
3/6 postgresql:setup / initdb_cache
OK 1.14s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
SKIP 0.18s
5/6 postgresql:xid_wraparound / xid_wraparound/002_limits
SKIP 0.19s
6/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
SKIP 0.19s
... snip
Providing PG_TEST_EXTRA as a configuration option works as well. But I
find it confusing
$ mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.71s
2/6 postgresql:setup / install_test_files
OK 0.06s
3/6 postgresql:setup / initdb_cache
OK 1.08s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
OK 52.73s 7 subtests passed
5/6 postgresql:xid_wraparound / xid_wraparound/002_limits
OK 56.36s 3 subtests passed
6/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
OK 160.46s 1 subtests passed
... snip
$ PG_TEST_EXTRA=ldap mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.37s
2/6 postgresql:setup / install_test_files
OK 0.08s
3/6 postgresql:setup / initdb_cache
OK 1.16s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
SKIP 0.14s
5/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
SKIP 0.15s
6/6 postgresql:xid_wraparound / xid_wraparound/002_limits
SKIP 0.15s
... snip
$ PG_TEST_EXTRA=xid_wraparound mts xid_wraparound
... snip
1/6 postgresql:setup / tmp_install
OK 0.36s
2/6 postgresql:setup / install_test_files
OK 0.06s
3/6 postgresql:setup / initdb_cache
OK 1.12s
4/6 postgresql:xid_wraparound / xid_wraparound/001_emergency_vacuum
OK 62.53s 7 subtests passed
5/6 postgresql:xid_wraparound / xid_wraparound/002_limits
OK 69.78s 3 subtests passed
6/6 postgresql:xid_wraparound / xid_wraparound/003_wraparounds
OK 186.78s 1 subtests passed
xid_wraparound tests are run if PG_TEST_EXTRA contains xid_wraparound
or it is not set. Any other setting will not run xid_wraparound test.
That's how the patch is coded but it isn't intuitive that changing
whether a test is run by default would require configuring the build
again. Probably we should just get rid of config time PG_TEST_EXTRA
altogether.
I am including +Tristan Partin who knows meson better.
If you are willing to work on this further, please add it to the commitfest.
--
Best Wishes,
Ashutosh Bapat
Hi, On Wed, 17 Jul 2024 at 13:13, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > xid_wraparound tests are run if PG_TEST_EXTRA contains xid_wraparound > or it is not set. Any other setting will not run xid_wraparound test. > That's how the patch is coded but it isn't intuitive that changing > whether a test is run by default would require configuring the build > again. Probably we should just get rid of config time PG_TEST_EXTRA > altogether. > > I am including +Tristan Partin who knows meson better. > > If you are willing to work on this further, please add it to the commitfest. I think I know why there is confusion. Could you try to set PG_TEST_EXTRA with quotes? Like PG_TEST_EXTRA="ldap mts xid_wraparound". -- Regards, Nazir Bilal Yavuz Microsoft
Hi, On Wed, 17 Jul 2024 at 13:23, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Hi, > > On Wed, 17 Jul 2024 at 13:13, Ashutosh Bapat > <ashutosh.bapat.oss@gmail.com> wrote: > > xid_wraparound tests are run if PG_TEST_EXTRA contains xid_wraparound > > or it is not set. Any other setting will not run xid_wraparound test. > > That's how the patch is coded but it isn't intuitive that changing > > whether a test is run by default would require configuring the build > > again. Probably we should just get rid of config time PG_TEST_EXTRA > > altogether. > > > > I am including +Tristan Partin who knows meson better. > > > > If you are willing to work on this further, please add it to the commitfest. > > I think I know why there is confusion. Could you try to set > PG_TEST_EXTRA with quotes? Like PG_TEST_EXTRA="ldap mts > xid_wraparound". Sorry, the previous reply was wrong; I misunderstood what you said. Yes, that is how the patch was coded and I agree that getting rid of config time PG_TEST_EXTRA could be a better alternative. -- Regards, Nazir Bilal Yavuz Microsoft
On Wed, Jul 17, 2024 at 3:34 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > Sorry, the previous reply was wrong; I misunderstood what you said. > Yes, that is how the patch was coded and I agree that getting rid of > config time PG_TEST_EXTRA could be a better alternative. Personally I use the config-time PG_TEST_EXTRA extensively. I'd be sad to see it go, especially if developers are no longer forced to use it. (In practice, I don't change that setting much after initial configure, because I use separate worktrees/build directories for different patchsets. And the reconfiguration is fast when I do need to modify it.) Thanks, --Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
> On Wed, Jul 17, 2024 at 3:34 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>> Sorry, the previous reply was wrong; I misunderstood what you said.
>> Yes, that is how the patch was coded and I agree that getting rid of
>> config time PG_TEST_EXTRA could be a better alternative.
> Personally I use the config-time PG_TEST_EXTRA extensively. I'd be sad
> to see it go, especially if developers are no longer forced to use it.
The existing and documented expectation is that PG_TEST_EXTRA is an
environment variable, ie it's a runtime option not a configure option.
Making it be the latter seems like a significant loss of flexibility
to me.
regards, tom lane
On 2024-07-17 We 11:01 AM, Tom Lane wrote:
Jacob Champion <jacob.champion@enterprisedb.com> writes:On Wed, Jul 17, 2024 at 3:34 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:Sorry, the previous reply was wrong; I misunderstood what you said. Yes, that is how the patch was coded and I agree that getting rid of config time PG_TEST_EXTRA could be a better alternative.Personally I use the config-time PG_TEST_EXTRA extensively. I'd be sad to see it go, especially if developers are no longer forced to use it.The existing and documented expectation is that PG_TEST_EXTRA is an environment variable, ie it's a runtime option not a configure option. Making it be the latter seems like a significant loss of flexibility to me.
AIUI the only reason we have it as a configure option at all is that meson is *very* dogmatic about not using environment variables. I get their POV when it comes to building, but that should not extend to testing. That said, I don't mind if this is a configure option as long as it can be overridden at run time without having to run "meson configure".
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Wed, Jul 17, 2024 at 8:01 AM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Jacob Champion <jacob.champion@enterprisedb.com> writes: > > Personally I use the config-time PG_TEST_EXTRA extensively. I'd be sad > > to see it go, especially if developers are no longer forced to use it. > > The existing and documented expectation is that PG_TEST_EXTRA is an > environment variable, ie it's a runtime option not a configure option. > Making it be the latter seems like a significant loss of flexibility > to me. I think/hope we're saying the same thing -- developers should not be forced to lock PG_TEST_EXTRA into their configurations; that's inflexible and unhelpful. What I'm saying in addition to that is, I really like that I can currently put a default PG_TEST_EXTRA into my meson config so that I don't have to keep track of it, and I do that all the time. So I'm in favor of the "option 3" approach. Thanks, --Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
> On Wed, Jul 17, 2024 at 8:01 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The existing and documented expectation is that PG_TEST_EXTRA is an
>> environment variable, ie it's a runtime option not a configure option.
>> Making it be the latter seems like a significant loss of flexibility
>> to me.
> I think/hope we're saying the same thing -- developers should not be
> forced to lock PG_TEST_EXTRA into their configurations; that's
> inflexible and unhelpful.
Indeed.
> What I'm saying in addition to that is, I really like that I can
> currently put a default PG_TEST_EXTRA into my meson config so that I
> don't have to keep track of it, and I do that all the time. So I'm in
> favor of the "option 3" approach.
Ah. I have no particular objection to that, but I wonder whether
we should make the autoconf/makefile infrastructure do it too.
regards, tom lane
On Wed, Jul 17, 2024 at 11:11 AM Tom Lane <tgl@sss.pgh.pa.us> wrote: > Ah. I have no particular objection to that, but I wonder whether > we should make the autoconf/makefile infrastructure do it too. I don't need it personally, having moved almost entirely to Meson. But if the asymmetry is a sticking point, I can work on a patch. Thanks! --Jacob
Hi, On Wed, 17 Jul 2024 at 13:13, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > If you are willing to work on this further, please add it to the commitfest. Since general consensus is more towards having an environment variable to override Meson configure option, I converted solution-3 to something more like a patch. I updated the docs, added more comments and added this to the commitfest [1]. The one downside of this approach is that PG_TEXT_EXTRA in user defined options in meson setup output could be misleading now. Also, with this change; PG_TEST_EXTRA from configure_scripts in the .cirrus.tasks.yml file should be removed as they are useless now. I added that as a second patch. [1] https://commitfest.postgresql.org/49/5134/ -- Regards, Nazir Bilal Yavuz Microsoft
Attachment
Hi Nazir, On Fri, Jul 19, 2024 at 1:37 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > > > If you are willing to work on this further, please add it to the commitfest. > > Since general consensus is more towards having an environment variable > to override Meson configure option, I converted solution-3 to > something more like a patch. I updated the docs, added more comments > and added this to the commitfest [1]. Thanks. > > The one downside of this approach is that PG_TEXT_EXTRA in user > defined options in meson setup output could be misleading now. > Upthread Tom asked whether we should do a symmetric change to "make". This set of patches does not do that. Here are my thoughts: 1. Those who use make, are not using configure time PG_TEST_EXTRA anyway, so they don't need it. 2. Those meson users who use setup time PG_TEST_EXTRA and also want to use make may find the need for the feature in make. 3. https://www.postgresql.org/docs/devel/install-requirements.html says that the meson support is currently experimental and only works when building from a Git checkout. So there's a possibility (even if theoretical) that make and meson will co-exist. Also that we may abandon meson? Considering those, it seems to me that symmetry is required. I don't know how hard it is to introduce PG_TEST_EXTRA as a configure time option in "make". If it's simple, we should do that. Otherwise it will be better to just remove PG_EXTRA_TEST option support from meson support to keep make and meson symmetrical. As far as the implementation is concerned the patch seems to be doing what's expected. If PG_TEST_EXTRA is specified at setup time, it is not needed to be specified as an environment variable at run time. But it can also be overridden at runtime. If PG_TEST_EXTRA is not specified at the time of setup, but specified at run time, it works. I have tested xid_wraparound and wal_consistency_check. I wonder whether we really require pg_test_extra argument to testwrap. Why can't we use the logic in testwrap, to set run time PG_TEST_EXTRA, in meson.build directly? I.e. set test_env['PG_TEST_EXTRA'] to os.environ[;PG_TEST_EXTRA'] if the latter is set, otherwise set the first to get_option('PG_TEST_EXTRA'). > Also, with this change; PG_TEST_EXTRA from configure_scripts in the > .cirrus.tasks.yml file should be removed as they are useless now. I > added that as a second patch. I think this is useful and allows both make and meson to use the same logic in cirrus. -- Best Wishes, Ashutosh Bapat
Hi, On Tue, 23 Jul 2024 at 12:26, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > Upthread Tom asked whether we should do a symmetric change to "make". > This set of patches does not do that. Here are my thoughts: > 1. Those who use make, are not using configure time PG_TEST_EXTRA > anyway, so they don't need it. > 2. Those meson users who use setup time PG_TEST_EXTRA and also want to > use make may find the need for the feature in make. > 3. https://www.postgresql.org/docs/devel/install-requirements.html > says that the meson support is currently experimental and only works > when building from a Git checkout. So there's a possibility (even if > theoretical) that make and meson will co-exist. Also that we may > abandon meson? > > Considering those, it seems to me that symmetry is required. I don't > know how hard it is to introduce PG_TEST_EXTRA as a configure time > option in "make". If it's simple, we should do that. Otherwise it will > be better to just remove PG_EXTRA_TEST option support from meson > support to keep make and meson symmetrical. I agree that symmetry should be the ultimate goal. Upthread Jacob said he could work on a patch about introducing the PG_TEST_EXTRA configure option to make builds. Would you still be interested in working on this? If not, I would gladly work on it. > As far as the implementation is concerned the patch seems to be doing > what's expected. If PG_TEST_EXTRA is specified at setup time, it is > not needed to be specified as an environment variable at run time. But > it can also be overridden at runtime. If PG_TEST_EXTRA is not > specified at the time of setup, but specified at run time, it works. I > have tested xid_wraparound and wal_consistency_check. Thanks for testing it! > I wonder whether we really require pg_test_extra argument to testwrap. > Why can't we use the logic in testwrap, to set run time PG_TEST_EXTRA, > in meson.build directly? I.e. set test_env['PG_TEST_EXTRA'] to > os.environ[;PG_TEST_EXTRA'] if the latter is set, otherwise set the > first to get_option('PG_TEST_EXTRA'). When test_env('PG_TEST_EXTRA') is set, it could not be overridden afterwards. Perhaps there is a way to override test_env() but I do not know how. -- Regards, Nazir Bilal Yavuz Microsoft
On Tue, Jul 23, 2024 at 4:02 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> > I wonder whether we really require pg_test_extra argument to testwrap.
> > Why can't we use the logic in testwrap, to set run time PG_TEST_EXTRA,
> > in meson.build directly? I.e. set test_env['PG_TEST_EXTRA'] to
> > os.environ[;PG_TEST_EXTRA'] if the latter is set, otherwise set the
> > first to get_option('PG_TEST_EXTRA').
>
> When test_env('PG_TEST_EXTRA') is set, it could not be overridden
> afterwards. Perhaps there is a way to override test_env() but I do not
> know how.
>
I am not suggesting to override test_env['PG_TEST_EXTRA'] but set it
to the value after overriding if required. meson.build file seems to
allow some conditional variable setting. So I thought this would be
possible, haven't tried myself though.
--
Best Wishes,
Ashutosh Bapat
Hi,
On Tue, 23 Jul 2024 at 13:40, Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Jul 23, 2024 at 4:02 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> >
> > > I wonder whether we really require pg_test_extra argument to testwrap.
> > > Why can't we use the logic in testwrap, to set run time PG_TEST_EXTRA,
> > > in meson.build directly? I.e. set test_env['PG_TEST_EXTRA'] to
> > > os.environ[;PG_TEST_EXTRA'] if the latter is set, otherwise set the
> > > first to get_option('PG_TEST_EXTRA').
> >
> > When test_env('PG_TEST_EXTRA') is set, it could not be overridden
> > afterwards. Perhaps there is a way to override test_env() but I do not
> > know how.
> >
>
> I am not suggesting to override test_env['PG_TEST_EXTRA'] but set it
> to the value after overriding if required. meson.build file seems to
> allow some conditional variable setting. So I thought this would be
> possible, haven't tried myself though.
Sorry if I caused a misunderstanding. What I meant was, when the
test_env('PG_TEST_EXTRA') is set, Meson will always use PG_TEST_EXTRA
from the setup. So, Meson needs to be built again to change
PG_TEST_EXTRA.
AFAIK Meson does not support accessing environment variables but
run_command() could be used to test this:
-test_env.set('PG_TEST_EXTRA', get_option('PG_TEST_EXTRA'))
+pg_test_extra_env = run_command(
+ [python,
+ '-c',
+ 'import os; print(os.getenv("PG_TEST_EXTRA", ""))'],
+ check: true).stdout()
+
+test_env.set('PG_TEST_EXTRA', pg_test_extra_env != '' ?
+ pg_test_extra_env :
+ get_option('PG_TEST_EXTRA'))
+
--
Regards,
Nazir Bilal Yavuz
Microsoft
On Tue, Jul 23, 2024 at 3:32 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > Upthread Jacob said he could work on a patch about introducing the > PG_TEST_EXTRA configure option to make builds. Would you still be > interested in working on this? If not, I would gladly work on it. Sure! Attached is a minimalist approach using AC_ARG_VAR. It works for top-level `make check-world`, or `make check -C src/test`. If you run `make check` directly from a test subdirectory, the variable doesn't get picked up, because it's only exported from the src/test level as of your patch c3382a3c3cc -- but if that turns out to be a problem, we can plumb it all the way down or expand the scope of the export. Thanks, --Jacob
Attachment
Hi Jacob, On Tue, Jul 23, 2024 at 7:54 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > > On Tue, Jul 23, 2024 at 3:32 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > Upthread Jacob said he could work on a patch about introducing the > > PG_TEST_EXTRA configure option to make builds. Would you still be > > interested in working on this? If not, I would gladly work on it. > > Sure! Attached is a minimalist approach using AC_ARG_VAR. > > It works for top-level `make check-world`, or `make check -C > src/test`. If you run `make check` directly from a test subdirectory, > the variable doesn't get picked up, because it's only exported from > the src/test level as of your patch c3382a3c3cc -- but if that turns > out to be a problem, we can plumb it all the way down or expand the > scope of the export. Sorry for the delay in reply. Here are my observations with the patch applied 1. If I run configure without setting PG_TEST_EXTRA, it won't run the tests that require PG_TEST_EXTRA to be set. This is expected. 2. But it wont' run tests even if PG_TEST_EXTRA is set while running make check.- that's unexpected 3. If I run configure with PG_TEST_EXTRA set and run 'make check' in the test directory, it runs those tests. That's expected from the final patch but that doesn't seem to be what you described above. 3. After 3, if I run `PG_TEST_EXTRA="something" make check`, it still runs those tests. So it looks like PG_TEST_EXTRA doesn't get overridden. If PG_TEST_EXTRA is set to something other than what was configured, it doesn't take effect when running the corresponding tests. E.g. PG_TEST_EXTRA is set to xid_wraparound at configure time, but `PG_TEST_EXTRA=wal_consistency_check make check ` is run, the tests won't use wal_consistency_check=all. - that's not expected. I this the patch lacks overriding PG_TEST_EXTRA at run time. AFAIU, following was expected behaviour from both meson and make. Please correct if I am wrong. 1. If PG_TEST_EXTRA is set at the setup/configuration time, it is not required to be set at run time. 2. Runtime PG_TEST_EXTRA always overrides configure time PG_TEST_EXTRA. -- Best Wishes, Ashutosh Bapat
On 2024-08-09 Fr 5:26 AM, Ashutosh Bapat wrote:
I this the patch lacks overriding PG_TEST_EXTRA at run time. AFAIU, following was expected behaviour from both meson and make. Please correct if I am wrong. 1. If PG_TEST_EXTRA is set at the setup/configuration time, it is not required to be set at run time. 2. Runtime PG_TEST_EXTRA always overrides configure time PG_TEST_EXTRA.
Yes, that's my understanding of the requirement.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Fri, Aug 9, 2024 at 2:26 AM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
> Here are my observations with the patch applied
> 1. If I run configure without setting PG_TEST_EXTRA, it won't run the
> tests that require PG_TEST_EXTRA to be set. This is expected.
> 2. But it wont' run tests even if PG_TEST_EXTRA is set while running
> make check.- that's unexpected
(see below)
> 3. If I run configure with PG_TEST_EXTRA set and run 'make check' in
> the test directory, it runs those tests. That's expected from the
> final patch but that doesn't seem to be what you described above.
I'm not entirely sure what you mean? src/test should work fine,
anything lower than that (say src/test/ssl) does not.
> 3. After 3, if I run `PG_TEST_EXTRA="something" make check`, it still
> runs those tests. So it looks like PG_TEST_EXTRA doesn't get
> overridden. If PG_TEST_EXTRA is set to something other than what was
> configured, it doesn't take effect when running the corresponding
> tests. E.g. PG_TEST_EXTRA is set to xid_wraparound at configure time,
> but `PG_TEST_EXTRA=wal_consistency_check make check ` is run, the
> tests won't use wal_consistency_check=all. - that's not expected.
I think you're running into the GNU Make override order [1]. For
instance when I want to override PG_TEST_EXTRA, I write
make check PG_TEST_EXTRA=whatever
If you want the environment variable to work by default instead, you can do
PG_TEST_EXTRA=whatever make check -e
If you don't want devs to have to worry about the difference, I think
we can change the assignment operator to `?=` in Makefile.global.in.
Thanks,
--Jacob
[1] https://www.gnu.org/software/make/manual/html_node/Environment.html
On Wed, Aug 14, 2024 at 2:24 AM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > > On Fri, Aug 9, 2024 at 2:26 AM Ashutosh Bapat > <ashutosh.bapat.oss@gmail.com> wrote: > > Here are my observations with the patch applied > > 1. If I run configure without setting PG_TEST_EXTRA, it won't run the > > tests that require PG_TEST_EXTRA to be set. This is expected. > > 2. But it wont' run tests even if PG_TEST_EXTRA is set while running > > make check.- that's unexpected > > (see below) > > > 3. If I run configure with PG_TEST_EXTRA set and run 'make check' in > > the test directory, it runs those tests. That's expected from the > > final patch but that doesn't seem to be what you described above. > > I'm not entirely sure what you mean? src/test should work fine, > anything lower than that (say src/test/ssl) does not. I could run them from src/test/modules/xid_wraparound/. That's desirable. > > > 3. After 3, if I run `PG_TEST_EXTRA="something" make check`, it still > > runs those tests. So it looks like PG_TEST_EXTRA doesn't get > > overridden. If PG_TEST_EXTRA is set to something other than what was > > configured, it doesn't take effect when running the corresponding > > tests. E.g. PG_TEST_EXTRA is set to xid_wraparound at configure time, > > but `PG_TEST_EXTRA=wal_consistency_check make check ` is run, the > > tests won't use wal_consistency_check=all. - that's not expected. > > I think you're running into the GNU Make override order [1]. For > instance when I want to override PG_TEST_EXTRA, I write > > make check PG_TEST_EXTRA=whatever > > If you want the environment variable to work by default instead, you can do > > PG_TEST_EXTRA=whatever make check -e > > If you don't want devs to have to worry about the difference, I think > we can change the assignment operator to `?=` in Makefile.global.in. What is working now should continue to work even after this change. PG_TEST_EXTRA="xyz" make check works right now. -- Best Wishes, Ashutosh Bapat
On Fri, Aug 23, 2024 at 5:59 AM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
> If I run
> export PG_TEST_EXTRA=xid_wraparound; ./configure --prefix=$BuildDir
> --enable-tap-tests && make -j4 && make -j4 install; unset
> PG_TEST_EXTRA
> followed by
> make -C $XID_MODULE_DIR check where
> XID_MODULE_DIR=src/test/modules/xid_wraparound - it skips the tests.
Right.
> I thought this was working before.
No, that goes back to my note in [1] -- as of c3382a3c3cc, the
variable was exported at only the src/test level, and I wanted to get
input on that so we could decide on the next approach if needed.
> Anyway, now I have written a script to test all the scenarios. You may
> want to test your patch using the script. It just needs PGDir to set
> to root directory of code.
Thanks! I see failures in 110, 120, and 130, as expected. Note that
constructions like
PG_TEST_EXTRA="" cd $XID_MODULE_DIR && make check && cd $PGDir
will not override the environment variable for the make invocation,
just for the `cd`. Also, rather than
export PG_TEST_EXTRA; ./configure ...; unset PG_TEST_EXTRA
it's probably easier to just pass PG_TEST_EXTRA=<setting> as a command
line option to configure.
> If there's some other way to setting PG_TEST_EXTRA when running
> configure, I think it needs to be documented.
./configure --help shows the new variable, with the same wording as
Meson. Or do you mean that it's significant enough to deserve a spot
in installation.sgml?
--Jacob
[1] https://www.postgresql.org/message-id/CAOYmi%2B%3D8HVgxANzFT_BZrAeDPxAgA5_kbHy-4VowdbGr0chHvQ%40mail.gmail.com
On Wed, Aug 28, 2024 at 5:26 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > On Fri, Aug 23, 2024 at 9:55 PM Jacob Champion > <jacob.champion@enterprisedb.com> wrote: > > > > On Fri, Aug 23, 2024 at 5:59 AM Ashutosh Bapat > > <ashutosh.bapat.oss@gmail.com> wrote: > > > If I run > > > export PG_TEST_EXTRA=xid_wraparound; ./configure --prefix=$BuildDir > > > --enable-tap-tests && make -j4 && make -j4 install; unset > > > PG_TEST_EXTRA > > > followed by > > > make -C $XID_MODULE_DIR check where > > > XID_MODULE_DIR=src/test/modules/xid_wraparound - it skips the tests. > > > > Right. > > > > > I thought this was working before. > > > > No, that goes back to my note in [1] -- as of c3382a3c3cc, the > > variable was exported at only the src/test level, and I wanted to get > > input on that so we could decide on the next approach if needed. I had an offline discussion with Jacob. Because of c3382a3c3cc, `make -C src/test/modules/xid_wraparound check` will skip xid_wraparound tests. It should be noted that when that commit was added well before the xid_wraparound tests were added. But I don't know whether the tests controlled by PG_TEST_EXTRA could be run using make -C that time. Anyway, I think, supporting PG_TEST_EXTRA at configure time without being able to run tests using `make -C <test directory> ` is not that useful. It only helps make check-world but not when running individual tests. Nazir, since you authored c3382a3c3cc, can you please provide input that Jacob needs? Otherwise, we should just go ahead with meson support and drop make support for now. We may revisit it later. -- Best Wishes, Ashutosh Bapat
On Wed, Aug 28, 2024 at 6:41 AM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > Nazir, since you authored c3382a3c3cc, can you please provide input > that Jacob needs? Specifically, why the PG_TEST_EXTRA variable was being exported at the src/test level only. If there's no longer a use case being targeted, we can always change it in this patch, but I didn't want to do that without understanding why it was like that to begin with. --Jacob
Hi, On Wed, 28 Aug 2024 at 16:41, Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > On Wed, Aug 28, 2024 at 5:26 PM Ashutosh Bapat > <ashutosh.bapat.oss@gmail.com> wrote: > > > > On Fri, Aug 23, 2024 at 9:55 PM Jacob Champion > > <jacob.champion@enterprisedb.com> wrote: > > > > > > On Fri, Aug 23, 2024 at 5:59 AM Ashutosh Bapat > > > <ashutosh.bapat.oss@gmail.com> wrote: > > > > If I run > > > > export PG_TEST_EXTRA=xid_wraparound; ./configure --prefix=$BuildDir > > > > --enable-tap-tests && make -j4 && make -j4 install; unset > > > > PG_TEST_EXTRA > > > > followed by > > > > make -C $XID_MODULE_DIR check where > > > > XID_MODULE_DIR=src/test/modules/xid_wraparound - it skips the tests. > > > > > > Right. > > > > > > > I thought this was working before. > > > > > > No, that goes back to my note in [1] -- as of c3382a3c3cc, the > > > variable was exported at only the src/test level, and I wanted to get > > > input on that so we could decide on the next approach if needed. > > I had an offline discussion with Jacob. Because of c3382a3c3cc, `make > -C src/test/modules/xid_wraparound check` will skip xid_wraparound > tests. It should be noted that when that commit was added well before > the xid_wraparound tests were added. But I don't know whether the > tests controlled by PG_TEST_EXTRA could be run using make -C that > time. I did not test but I think they could not be run because of the same reason as below. > Anyway, I think, supporting PG_TEST_EXTRA at configure time without > being able to run tests using `make -C <test directory> ` is not that > useful. It only helps make check-world but not when running individual > tests. > > Nazir, since you authored c3382a3c3cc, can you please provide input > that Jacob needs? I think the problem is that we define PG_TEST_EXTRA in the Makefile.global file but we do not export it there, instead we export in the src/test/Makefile. But when you run tests from their Makefiles, they do not include src/test/Makefile (they include Makefile.global); so they can not get the PG_TEST_EXTRA env variable. Exporting PG_TEST_EXTRA in the Makefile.global should solve the problem. Also, I think TEST 110 and 170 do not look correct to me. In the current way, we do not pass PG_TEST_EXTRA to the make command. 110 should be: 'cd $XID_MODULE_DIR && PG_TEST_EXTRA=xid_wraparound make check' instead of 'PG_TEST_EXTRA=xid_wraparound cd $XID_MODULE_DIR && make check' 170 should be: 'cd $XID_MODULE_DIR && PG_TEST_EXTRA="" make check && cd $PGDir' instead of 'PG_TEST_EXTRA="" cd $XID_MODULE_DIR && make check && cd $PGDir' -- Regards, Nazir Bilal Yavuz Microsoft
Hi, On Wed, 28 Aug 2024 at 18:11, Jacob Champion <jacob.champion@enterprisedb.com> wrote: > > On Wed, Aug 28, 2024 at 6:41 AM Ashutosh Bapat > <ashutosh.bapat.oss@gmail.com> wrote: > > Nazir, since you authored c3382a3c3cc, can you please provide input > > that Jacob needs? > > Specifically, why the PG_TEST_EXTRA variable was being exported at the > src/test level only. If there's no longer a use case being targeted, > we can always change it in this patch, but I didn't want to do that > without understanding why it was like that to begin with. I do not exactly remember the reason but I think I copied the same behavior as before, PG_TEST_EXTRA variable was checked in the src/test/Makefile so I exported it there. -- Regards, Nazir Bilal Yavuz Microsoft
Hi, On Fri, 30 Aug 2024 at 21:36, Jacob Champion <jacob.champion@enterprisedb.com> wrote: > > On Wed, Aug 28, 2024 at 8:21 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: > > I do not exactly remember the reason but I think I copied the same > > behavior as before, PG_TEST_EXTRA variable was checked in the > > src/test/Makefile so I exported it there. > > Okay, give v3 a try then. This exports directly from Makefile.global. > Since that gets pulled into a bunch of places, the scope is a lot > wider than it used to be; I've disabled it for PGXS so it doesn't end > up poisoning other extensions. Patch looks good and it passes all the test cases in Ashutosh's test script. -- Regards, Nazir Bilal Yavuz Microsoft
On Mon, Sep 2, 2024 at 8:32 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
>
> Hi,
>
> On Fri, 30 Aug 2024 at 21:36, Jacob Champion
> <jacob.champion@enterprisedb.com> wrote:
> >
> > On Wed, Aug 28, 2024 at 8:21 AM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:
> > > I do not exactly remember the reason but I think I copied the same
> > > behavior as before, PG_TEST_EXTRA variable was checked in the
> > > src/test/Makefile so I exported it there.
> >
> > Okay, give v3 a try then. This exports directly from Makefile.global.
> > Since that gets pulled into a bunch of places, the scope is a lot
> > wider than it used to be;
This looks similar to what meson does with PG_TEST_EXTRA, it is
available via get_option(). So we are closing the gap between meson
and make. That's the intention.
> I've disabled it for PGXS so it doesn't end
> > up poisoning other extensions.
>
> Patch looks good and it passes all the test cases in Ashutosh's test script.
>
Thanks Bilal for testing the patch. Can you or Jacob please create one
patchset including both meson and make fixes? Please keep the meson
and make changes in separate patches though. I think the meson patches
come from [1] (they probably need a rebase, git am failed) and make
patch comes from [2].The one fixing make needs a good commit message.
some comments on code
1. comments on changes to meson
+ variable if it exists. See <xref linkend="regress-additional"/> for
s/exists/set/
-# Test suites that are not safe by default but can be run if selected
-# by the user via the whitespace-separated list in variable PG_TEST_EXTRA.
-# Export PG_TEST_EXTRA so it can be checked in individual tap tests.
-test_env.set('PG_TEST_EXTRA', get_option('PG_TEST_EXTRA'))
A naive question. What happens if we add PG_TEST_EXTRA in meson.build
itself rather than passing it via testwrap? E.g. like
if "PG_TEST_EXTRA" not in os.environ
test_env.set('PG_TEST_EXTRA', get_option('PG_TEST_EXTRA'))
I am worried that we might have to add an extra argument to testwrap
for every environment variable that influences the tests. Avoiding it
would be better.
option('PG_TEST_EXTRA', type: 'string', value: '',
- description: 'Enable selected extra tests')
+ description: 'Enable selected extra tests, please note that this
configure option is overridden by PG_TEST_EXTRA environment variable
if it exists')
All the descriptions look much shorter than this one. I suggest we
shorten this one too as
"Enable selected extra tests. Overridden by PG_TEST_EXTRA environment variable."
not as short as other descriptions but shorter than before and yet
serves its intended purpose. Or just make it the same as the one in
configure.ac. Either way the descriptions in configure.ac and
meson_options.txt should be in sync.
+# If PG_TEST_EXTRA is not set in the environment, then look for its Meson
+# configure option.
+if "PG_TEST_EXTRA" not in os.environ and args.pg_test_extra:
+ env_dict["PG_TEST_EXTRA"] = args.pg_test_extra
+
If somebody looks at just these lines, it's not clear how
PG_TEST_EXTRA is passed to the test environment if it's available in
os.environ. So one has to understand that env_dict is the test
environment. If that's so, the code and comment rewritten as below
makes more sense to me. What do you think?
# If PG_TEST_EXTRA is not already part of the test environment, check if it's
# passed via program argument --pg_test_extra. Thus we also override
# configuration time value by run time value of PG_TEST_EXTRA.
if "PG_TEST_EXTRA" not in env_dict and args.pg_test_extra:
env_dict["PG_TEST_EXTRA"] = args.pg_test_extra
But in case we decide to fix meson.build as suggested in one of the
commentsabove, this change will be unnecessary.
Note that PG_TEST_EXTRA is used in only TAP tests right now. The way
the value passed to --pg_test_extra is handled in testwrap, it will
available to every test, not just TAP tests. But this looks fine to me
since the documentation of PG_TEST_EXTRA or its name itself does not
show any intention to limit it only to TAP tests.
2. comments on make changes
Since Makefile.global.in is included in src/test/Makefile I was
expecting that the PG_TEST_EXTRA picked from configuration would be
available in src/test/Makefile from which it would be exported. But
that doesn't seem to be the case. In fact, I am getting doubtful about
the role of the current "export PG_TEST_EXTRA" in /src/test/Makefile.
Even if I remove it, it doesn't affect anything. Commands a.
PG_TEST_EXTRA=xid_wraparound make check, b.
PG_TEST_EXTRA=xid_wraparound make -C $XID_MODULE_DIR check run the
tests (and don't skip them).
Anyway with the proposed change PG_TEST_EXTRA passed at configuration
time is used if it's not defined at run time as expected. I think the
patch looks good. Nothing to complain there.
[1] https://www.postgresql.org/message-id/CAN55FZ1Tko2N=X4f6icgFhb7syJYo_APP-9EbFcT-uH6tEi_Xg@mail.gmail.com
[2] https://www.postgresql.org/message-id/CAOYmi+=6HNVhbFOVsV6X2_DVDYcUDL4AMnj7iM15gAfw__beKA@mail.gmail.com
--
Best Wishes,
Ashutosh Bapat
On 13/09/2024 13:41, Ashutosh Bapat wrote: > On Thu, Sep 12, 2024 at 4:28 PM Nazir Bilal Yavuz <byavuz81@gmail.com> wrote: >>> >>> Once this is done, I think we can mark this CF entry as RFC. >> >> Thanks for the changes. I applied all of them in respective patches. > > Thanks a lot. PFA the patchset with > > 1. Improved comment related to PG_TEST_EXTRA in meson.build. More on > the improvement in the commit message of patch 0002, which should be > merged into 0001. > 2. You have written comprehensive commit messages. I elaborated on > them a bit. I have left your version in the commit message for > committer to pick up appropriate one. > > Marking the entry as RFC. Committed with minor changes. I squashed the first two patches, and rephrased the docs and some comments a little. Thanks! -- Heikki Linnakangas Neon (https://neon.tech)