Thread: Contributing test cases to improve coverage
Hello All,
(a) The regression test suite is run by a parallel scheduler, with some test cases dependent on previous test cases. If I just add my test case as part of the parallel scheduler’s tests, it might not work, since previous test cases in the scheduler might already create the same table, for instance.
(b) How do I get my test cases reviewed and ultimately included in a future release of PostgreSQL?
I am working on a project that aims to produce test cases that improve mutation coverage of a dbms's test suite.
The rough workflow of the project goes as follows:
(a) apply mutation at a souce code level
(b) compile and check if the mutated installation passed existing testsuite
(c) If not, fuzz the mutated installation with SQL fuzzer
(d) if a fuzzor successfully produce a test case that crash or trigger bugs in the mutated installation, use a reduction tools to reduce the test case
(e) add the reduced test case to existing test suite
For postgres, I am looking at adding test cases to test suite in test/src/regress/. I have gone through (a)-(e), and managed to produced some test cases. As an example, I claim the test case
```
CREATE RECURSIVE VIEW a(b) AS SELECT'' ;
SELECT FROM a WHERE NULL;
```
could kill the following mutation at optimizer/plan/setrefs.c, 502:5--502:33
Original binary operator expression:
```
rte->rtekind == RTE_SUBQUERY
````
Replacement expression:
```
(rte->rtekind) >= RTE_SUBQUERY
rte->rtekind == RTE_SUBQUERY
````
Replacement expression:
```
(rte->rtekind) >= RTE_SUBQUERY
```
I have a few questions about adding these test cases:
(a) The regression test suite is run by a parallel scheduler, with some test cases dependent on previous test cases. If I just add my test case as part of the parallel scheduler’s tests, it might not work, since previous test cases in the scheduler might already create the same table, for instance.
(b) How do I get my test cases reviewed and ultimately included in a future release of PostgreSQL?
Thank you for your time.
Regards,
Jon
J F <jonathanfoo0523@gmail.com> writes:
> For postgres, I am looking at adding test cases to test suite in
> test/src/regress/. I have gone through (a)-(e), and managed to produced
> some test cases. As an example, I claim the test case
> ```
> CREATE RECURSIVE VIEW a(b) AS SELECT'' ;
> SELECT FROM a WHERE NULL;
> ```
> could kill the following mutation at optimizer/plan/setrefs.c, 502:5--502:33
> Original binary operator expression:
> ```
> rte->rtekind == RTE_SUBQUERY
> ````
> Replacement expression:
> ```
> (rte->rtekind) >= RTE_SUBQUERY
> ```
I am quite confused about what is the point of this. You have not
found any actual bug, nor have you demonstrated that this test case
could discover a likely future bug that wouldn't be detected another
way. Moreover, it seems like the process would lead to some very
large number of equally marginal test cases. We aren't likely to
accept such a patch, because we are concerned about keeping down the
runtime of the test suite.
regards, tom lane
> I am quite confused about what is the point of this. You have not> found any actual bug, nor have you demonstrated that this test case> could discover a likely future bug that wouldn't be detected another> way. Moreover, it seems like the process would lead to some very> large number of equally marginal test cases. We aren't likely to> accept such a patch, because we are concerned about keeping down the> runtime of the test suite.>> regards, tom lane
The point of this project is to improve the coverage of PostgreSQL’s preexisting test suite. Writing a test suite to achieve close to 100% coverage is challenging, but I have proposed a workflow to automate this process.
I assert that no test case in the regression test suite currently covers the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose adding a new test case that addresses exactly this. In the future, if someone accidentally modifies the operator to become >=, it will trigger incorrect behavior when certain queries are executed. This test case will catch that issue.
I get that the test cases in /regress are likely reserved for actual bugs found and are designed to run quickly. Would it be a good idea to have a separate, more rigorous test suite that runs longer but provides better code coverage?
Regards,
Jon
On Thu, Jun 13, 2024 at 1:22 AM J F <jonathanfoo0523@gmail.com> wrote:
> I am quite confused about what is the point of this. You have not> found any actual bug, nor have you demonstrated that this test case> could discover a likely future bug that wouldn't be detected another> way. Moreover, it seems like the process would lead to some very> large number of equally marginal test cases. We aren't likely to> accept such a patch, because we are concerned about keeping down the> runtime of the test suite.>> regards, tom laneThe point of this project is to improve the coverage of PostgreSQL’s preexisting test suite. Writing a test suite to achieve close to 100% coverage is challenging, but I have proposed a workflow to automate this process.
We monitor code coverage. But this is doing more than that. It will find out the places in code, which if changed, will cause bugs. That seems useful to avoid refactoring mistakes, esp. when people rely on regression tests to tell whether their code changes are sane. But in PostgreSQL, we rely heavily on reviewers and committers to do that instead of tests. Still, the tests produced by this tool will help catch bugs that human eyes can not. As Tom said, managing that battery of tests may not be worth it. Basically, I am flip-flopping on the usefulness of this effort.
I assert that no test case in the regression test suite currently covers the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose adding a new test case that addresses exactly this. In the future, if someone accidentally modifies the operator to become >=, it will trigger incorrect behavior when certain queries are executed. This test case will catch that issue.
Usually PostgreSQL developers know that rtekind is an enum so they are very very unlikely to use anything other than == and !=. Such a change will be caught by a reviewer. I think many of the tests that this tool will produce will fall in this category.
I get that the test cases in /regress are likely reserved for actual bugs found and are designed to run quickly. Would it be a good idea to have a separate, more rigorous test suite that runs longer but provides better code coverage?
There are practical difficulties like maintaining the expected outputs for such a large battery of tests. But maybe some external project could.
BTW, have you considered perl tests, isolation tests etc. Tests in regress/ do not cover many subsystems e.g. replication.
-- Best Wishes,
Ashutosh Bapat
On 12.06.24 18:44, J F wrote: > (a) The regression test suite is run by a parallel scheduler, with some > test cases dependent on previous test cases. If I just add my test case > as part of the parallel scheduler’s tests, it might not work, since > previous test cases in the scheduler might already create the same > table, for instance. Yes, you need to take care of that somehow. Some test files put all their test objects in a schema. Others are careful to drop all test objects at the end. Or you just have to pick non-conflicting names. > (b) How do I get my test cases reviewed and ultimately included in a > future release of PostgreSQL? Perhaps start with https://wiki.postgresql.org/wiki/Development_information and in particular https://wiki.postgresql.org/wiki/Submitting_a_Patch