Thread: tests and meson - test names and file locations
Hi, One of the things motivating me to work on the meson conversion is the ability to run tests in an easily understandable way. Meson has a testrunner that both allows to run all tests at once, and run subsets of tests. = Test and testsuite naming = Each test has a unique name, and 0-n labels (called 'suites'). One can run tests by their name, or select tests by suites. The way I've organized it so far is that tests are named like this: main/pg_regress main/isolation recovery/t/018_wal_optimize.pl pg_prewarm/t/001_basic.pl test_shm_mq/pg_regress psql/t/001_basic.pl we could also name tests by their full path, but that makes the display very unwieldy. At the moment there's three suites differentiating by the type of test: 'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites, describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc. That currently works out to each test having two suites, although I've wondered about adding a 'contrib' suite as well. Perhaps the pg_regress suite should just be named 'regress'? And perhaps the t/ in the tap tests is superfluous - the display is already fairly wide? = Log and Data locations = To make things like the selection of log files for a specific test easier, I've so far set it up so that test data and logs are stored in a separate directory from the sources. testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...> The runner now creates a test.start at the start of a test and either test.success or test.failure at the end. That should make it pretty easy for e.g. the buildfarm and CI to make the logs for a failed test easily accessible. I've spent far too much time going through the ~hundred logs in src/test/recovery/ that the buildfarm displays as one thing. I really like having all the test data separately from the sources, but I get that that's not what we've done so far. It's doable to just mirror the current choice, but I don't think we should. But I won't push too hard against keeping things the same. I do wonder if we should put test data and log files in a separate directory tree, but that'd be a bit more work probably. Any comments on the above? = Example outputs = Here's an example output that you mostly should be able to make sense of now: $ m test --print-errorlogs ninja: Entering directory `/tmp/meson' ninja: no work to do. 1/242 postgresql:setup / tmp_install OK 0.33s 2/242 postgresql:tap+pg_upgrade / pg_upgrade/t/001_basic.pl OK 0.35s 8 subtests passed 3/242 postgresql:tap+recovery / recovery/t/011_crash_recovery.pl OK 2.67s 3 subtests passed 4/242 postgresql:tap+recovery / recovery/t/013_crash_restart.pl OK 2.91s 18 subtests passed 5/242 postgresql:tap+recovery / recovery/t/014_unlogged_reinit.pl OK 3.01s 23 subtests passed 6/242 postgresql:tap+recovery / recovery/t/022_crash_temp_files.pl OK 3.16s 11 subtests passed 7/242 postgresql:tap+recovery / recovery/t/016_min_consistency.pl OK 3.43s 1 subtests passed 8/242 postgresql:tap+recovery / recovery/t/021_row_visibility.pl OK 3.46s 10 subtests passed 9/242 postgresql:isolation+tcn / tcn/isolation OK 3.42s 10/242 postgresql:tap+recovery / recovery/t/023_pitr_prepared_xact.pl OK 3.63s 1 subtests passed ... 241/242 postgresql:isolation+main / main/isolation OK 46.69s 242/242 postgresql:tap+pg_upgrade / pg_upgrade/t/002_pg_upgrade.pl OK 57.00s 13 subtests passed Ok: 242 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Full log written to /tmp/meson/meson-logs/testlog.txt The 'postgresql' is because meson supports subprojects (both to provide dependencies if needed, and "real" subprojects), and their tests can be run at once. If a test fails it'll show the error output at the time of test: 39/242 postgresql:pg_regress+cube / cube/pg_regress FAIL 3.74s exit status 1 >>> REGRESS_SHLIB=/tmp/meson/src/test/regress/regress.so MALLOC_PERTURB_=44 PG_TEST_EXTRA='kerberos ldap ssl' PG_REGRESS=/tmp/meson/src/test/regress/pg_regress PATH=/tmp/meson/tmp_install/tmp/meson-install/bin:/tmp/meson/contrib/cube:/home/andres/bin/perl5/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games /usr/bin/python3/home/andres/src/postgresql/src/tools/testwrap --srcdir /home/andres/src/postgresql/contrib/cube --basedir/tmp/meson --builddir /tmp/meson/contrib/cube --testgroup cube --testname pg_regress /tmp/meson/src/test/regress/pg_regress--temp-instance /tmp/meson/testrun/cube/pg_regress/tmp_check --inputdir /home/andres/src/postgresql/contrib/cube--expecteddir /home/andres/src/postgresql/contrib/cube --outputdir /tmp/meson/testrun/cube/pg_regress--bindir '' --dlpath /tmp/meson/contrib/cube --max-concurrent-tests=20 --port=40012 cubecube_sci ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― # executing test in /tmp/meson/testrun/cube/pg_regress group cube test pg_regress, builddir /tmp/meson/contrib/cube ============== creating temporary instance ============== ============== initializing database system ============== ============== starting postmaster ============== running on port 40012 with PID 354981 ============== creating database "regression" ============== CREATE DATABASE ALTER DATABASE ALTER DATABASE ALTER DATABASE ALTER DATABASE ALTER DATABASE ALTER DATABASE ============== running regression test queries ============== test cube ... FAILED 418 ms test cube_sci ... ok 16 ms ============== shutting down postmaster ============== ====================== 1 of 2 tests failed. ====================== The differences that caused some tests to fail can be viewed in the file "/tmp/meson/testrun/cube/pg_regress/regression.diffs". A copy of the test summary that you see above is saved in the file "/tmp/meson/testrun/cube/pg_regress/regression.out". # test failed ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― 40/242 postgresql:pg_regress+citext / citext/pg_regress OK 3.77s ... A list of failed tests is listed at the end of the test: Summary of Failures: 39/242 postgresql:pg_regress+cube / cube/pg_regress FAIL 3.74s exitstatus 1 Ok: 241 Expected Fail: 0 Fail: 1 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Full log written to /tmp/meson/meson-logs/testlog.txt Greetings, Andres Freund
On 2022-08-11 Th 00:04, Andres Freund wrote: > > The runner now creates a test.start at the start of a test and either > test.success or test.failure at the end. That should make it pretty easy for > e.g. the buildfarm and CI to make the logs for a failed test easily > accessible. I've spent far too much time going through the ~hundred logs in > src/test/recovery/ that the buildfarm displays as one thing. I do have work in hand to improve that markedly, just need a little time to finish it. > > > I really like having all the test data separately from the sources, but I get > that that's not what we've done so far. It's doable to just mirror the current > choice, but I don't think we should. But I won't push too hard against keeping > things the same. I also like that. I think we should take this opportunity for some serious rationalization of this. Tests and associated data have grown rather like Topsy, and we should fix that. So please don't feel too constrained by current practice. > > > I do wonder if we should put test data and log files in a separate directory > tree, but that'd be a bit more work probably. > > > Any comments on the above? > > > = Example outputs = [...] > Full log written to /tmp/meson/meson-logs/testlog.txt /tmp ? cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
> I also like that. I think we should take this opportunity for some
> serious rationalization of this. Tests and associated data have grown
> rather like Topsy, and we should fix that. So please don't feel too
> constrained by current practice.
I'm definitely -1 on that. Major rearrangement of the test scripts
would be a huge blocking factor for almost any back-patch. I don't
care much if you want to rearrange how the tests are invoked, but
please don't touch the individual .sql and .pl scripts.
regards, tom lane
Hi, On 2022-08-11 10:06:35 -0400, Andrew Dunstan wrote: > > Full log written to /tmp/meson/meson-logs/testlog.txt > > /tmp ? I often put throwaway buildtrees in /tmp. So this is just because my buildtree is in /tmp/meson, i.e. the log always is in $build_root/meson-logs/testlog.txt (there's also a log of the configure run in there etc). Greetings, Andres Freund
Hi, On 2022-08-11 10:20:42 -0400, Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > > I also like that. I think we should take this opportunity for some > > serious rationalization of this. Tests and associated data have grown > > rather like Topsy, and we should fix that. So please don't feel too > > constrained by current practice. > > I'm definitely -1 on that. Major rearrangement of the test scripts > would be a huge blocking factor for almost any back-patch. I don't > care much if you want to rearrange how the tests are invoked, but > please don't touch the individual .sql and .pl scripts. I don't precisely know what Andrew was thinking of, but the relocation of log files for example doesn't require many changes to .pl files - one change to Utils.pm. The one exception to that is 010_tab_completion.pl, which encodes tmp_check/ in its output. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> I don't precisely know what Andrew was thinking of, but the relocation of log
> files for example doesn't require many changes to .pl files - one change to
> Utils.pm. The one exception to that is 010_tab_completion.pl, which encodes
> tmp_check/ in its output.
Ah. That seems perfectly tolerable. Andrew seemed to be thinking of
moving the tests' source files around.
regards, tom lane
Andres Freund <andres@anarazel.de> writes:
> = Log and Data locations =
> To make things like the selection of log files for a specific test easier,
> I've so far set it up so that test data and logs are stored in a separate
> directory from the sources.
> testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>
> I do wonder if we should put test data and log files in a separate directory
> tree, but that'd be a bit more work probably.
I'm confused, didn't you just say you already did that?
> Here's an example output that you mostly should be able to make sense of now:
TBH, this seems to be almost all the same sort of useless noise that
we have worked to suppress in "make" output. The only thing that's
of any interest at all is the report that the "cube" test failed,
and you have managed to make it so that that report is pretty
voluminous and yet contains not one useful detail. I still have
to go and look at other log files to figure out what happened;
and if I need to see the postmaster log, it's not even apparent
where that is. I also wonder where, say, a core dump might wind up.
I'm failing to see any advance at all here over what we have now.
If anything, the signal-to-noise ratio has gotten worse.
regards, tom lane
On 2022-08-11 Th 11:06, Andres Freund wrote: > Hi, > > On 2022-08-11 10:20:42 -0400, Tom Lane wrote: >> Andrew Dunstan <andrew@dunslane.net> writes: >>> I also like that. I think we should take this opportunity for some >>> serious rationalization of this. Tests and associated data have grown >>> rather like Topsy, and we should fix that. So please don't feel too >>> constrained by current practice. >> I'm definitely -1 on that. Major rearrangement of the test scripts >> would be a huge blocking factor for almost any back-patch. I don't >> care much if you want to rearrange how the tests are invoked, but >> please don't touch the individual .sql and .pl scripts. > I don't precisely know what Andrew was thinking of, but the relocation of log > files for example doesn't require many changes to .pl files - one change to > Utils.pm. The one exception to that is 010_tab_completion.pl, which encodes > tmp_check/ in its output. I meant test results, logs et. I thought that was the topic under discussion. Changing the location of test sources would be a whole other topic. Sorry if I was not clear enough. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2022-08-11 13:06:35 -0400, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > = Log and Data locations = > > > To make things like the selection of log files for a specific test easier, > > I've so far set it up so that test data and logs are stored in a separate > > directory from the sources. > > > testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...> > > > I do wonder if we should put test data and log files in a separate directory > > tree, but that'd be a bit more work probably. > > I'm confused, didn't you just say you already did that? I did separate the source / build tree from the test log files / data files, but not the test log files from the test data files. It'd be easy enough to achieve for pg_regress tests, but perhaps a bit harder for things like 027_stream_regress.pl that run pg_regress themselves. > > Here's an example output that you mostly should be able to make sense of now: > > TBH, this seems to be almost all the same sort of useless noise that > we have worked to suppress in "make" output. Which part have we suppressed that's shown here? We've been talking about cutting down the pointless information that pg_regress produces, but that seems like a separate endeavor. > The only thing that's of any interest at all is the report that the "cube" > test failed, and you have managed to make it so that that report is pretty > voluminous and yet contains not one useful detail. It'll just print the list of tests and their success / failure, without printing the test's output, if you don't pass --print-errorlogs. The reason the commmand is shown is so you can copy-paste it to run the tests on its own, which can be useful for debugging, it'll include all the environment variables set when the test was run, so it's actually the same command (not like right now, were some env variables are set via export in the makefile). We probably can make the command shorter - but that's pretty similar to what's done for the make invocation of the tests. > I still have to go and look at other log files to figure out what happened; > and if I need to see the postmaster log, it's not even apparent where that > is. There's a hint at the start, but it could stand to be reformulated to point that out more clearly: # executing test in /tmp/meson/testrun/cube/pg_regress group cube test pg_regress, builddir /tmp/meson/contrib/cube > I also wonder where, say, a core dump might wind up. That'll depend on system configuration as before. Most commonly in the data dir. Are you wondering where the data dir is? > I'm failing to see any advance at all here over what we have now. > If anything, the signal-to-noise ratio has gotten worse. I'm surprised - I find it *vastly* more readable, because it'll show this stuff only for the failed tests, and it'll tell you the failed tests at the end. I don't know how many hours I've spent going backward through check-world output to find the first failed tests, but it's many. Greetings, Andres Freund
On 11.08.22 06:04, Andres Freund wrote: > At the moment there's three suites differentiating by the type of test: > 'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites, > describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc. > > That currently works out to each test having two suites, although I've > wondered about adding a 'contrib' suite as well. I'm not sure what the value of these suite names would be. I don't usually find myself wanting to run, say, just all tap tests. Perhaps suites would be useful to do things like select slow tests (numeric_big) or handle the ssl, ldap, kerberos tests. > = Log and Data locations = > > To make things like the selection of log files for a specific test easier, > I've so far set it up so that test data and logs are stored in a separate > directory from the sources. > > testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...> > > The runner now creates a test.start at the start of a test and either > test.success or test.failure at the end. That should make it pretty easy for > e.g. the buildfarm and CI to make the logs for a failed test easily > accessible. I've spent far too much time going through the ~hundred logs in > src/test/recovery/ that the buildfarm displays as one thing. I don't really understand which problem this solves and how. Sure, the test output is somewhat complex, but I know where it is and I've never found myself wishing it to be somewhere else.
Hi, On 2022-08-12 18:08:00 +0200, Peter Eisentraut wrote: > > At the moment there's three suites differentiating by the type of test: > > 'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites, > > describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc. > > > > That currently works out to each test having two suites, although I've > > wondered about adding a 'contrib' suite as well. > > I'm not sure what the value of these suite names would be. I don't usually > find myself wanting to run, say, just all tap tests. I occasionally want to, but it may not be important enough. I do find it useful for display purposes alone, and for finding kinds of test with meson test --list. > On 11.08.22 06:04, Andres Freund wrote: > > = Log and Data locations = > > > > To make things like the selection of log files for a specific test easier, > > I've so far set it up so that test data and logs are stored in a separate > > directory from the sources. > > > > testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...> > > > > The runner now creates a test.start at the start of a test and either > > test.success or test.failure at the end. That should make it pretty easy for > > e.g. the buildfarm and CI to make the logs for a failed test easily > > accessible. I've spent far too much time going through the ~hundred logs in > > src/test/recovery/ that the buildfarm displays as one thing. > > I don't really understand which problem this solves and how. Sure, the test > output is somewhat complex, but I know where it is and I've never found > myself wishing it to be somewhere else. I'd like the buildfarm and CI a) use parallelism to run tests (that's why the BF is slow) b) show the logfiles for exactly the failed test ([1]). We can of course iterate through the whole directory tree, somehow identify which log files are for which test, and then select the log files for the failed tests. But that's much easier to do then when you have a uniform directory hierarchy, where you can test which tests have failed based on the filesystem alone. Greetings, Andres Freund [1] E.g. the log file for this failed run is 13MB, and I've had ones with a bit more debugging enabled crash both firefox and chrome before https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2022-08-06%2012%3A17%3A14&stg=recovery-check
On 12.08.22 18:29, Andres Freund wrote: >> I don't really understand which problem this solves and how. Sure, the test >> output is somewhat complex, but I know where it is and I've never found >> myself wishing it to be somewhere else. > I'd like the buildfarm and CI a) use parallelism to run tests (that's why the > BF is slow) b) show the logfiles for exactly the failed test ([1]). We can of > course iterate through the whole directory tree, somehow identify which log > files are for which test, and then select the log files for the failed > tests. But that's much easier to do then when you have a uniform directory > hierarchy, where you can test which tests have failed based on the filesystem > alone. My initial experiences with testing under meson is that it's quite fragile and confusing (unlike the building, which is quite robust and understandable). Some of that is the fault of meson, some of that is our implementation. Surely this can be improved over time, but my experience has been that it's not there yet. The idea that we are going to move all the test output files somewhere else at the same time is not appealing to me. The combination of fragile plus can't find the diagnostics is not a good one. Now, this is my experience; others might have different ones. Also, is there anything in these proposed changes that couldn't also be applied to the old build system? We are going to be running them in parallel for some time. It would be good if one doesn't have to learn two entirely different sets of testing interfaces.