Thread: On Judging the Value of Tests
Dear PG Experts,
I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced PG devs here could provide some insights on testing because I could not find concrete answers in the Postgres documentations.
1. How do you judge when a test suite is acceptable to be added to Postgres OSS source code? (How do you judge the value of a test suite?)
2. Besides code coverage test, are there other ways/techniques/tools to measure how much value a test would have?
This is coming from a perspective that in the current code coverage (https://coverage.postgresql.org/), some functions were covered because the function was called/run in some tests, while the function/lines may also not necessarily be thoroughly/directly tested so the coverage skips some lines of code inside a block.
3. Is there a standard way of writing tests on the source code that we should follow, like when should test be written in TAP/SQL/C formats and how long should it be?
① I know that TAP test is for client program tests, SQL for regression tests with sql queries, but there are instances also where tests are also written in C like isolation tests, etc. How do we best judge which language is preferred to use when writing tests for Postgres components? How long should a test be when proposing them to hackers page?
4. In the src/test/examples directory (which are all libpq tests), why is the “examples” directory not included when building postgres? (Why weren't these libpq tests added to src/interface/libpq/test or in regression test suite instead?) In short, how to know where (in which file/directory in source code) to put a test?
I’d like to apologize for the series of questions, but I’d really like to get insights from the experienced devs.
It's alright if you don't have answers to all questions as this might take much of your time.
Thank you very much.
Regards,
Kirk Jamison
Dear PG Experts,
I am entirely very new to PG development.
Currently, I’m studying the test suites, and I feel the more experienced PG devs here could provide some insights on testing because I could not find concrete answers in the Postgres documentations.
1. How do you judge when a test suite is acceptable to be added to Postgres OSS source code? (How do you judge the value of a test suite?)
3. Is there a standard way of writing tests on the source code that we should follow, like when should test be written in TAP/SQL/C formats and how long should it be?
① I know that TAP test is for client program tests, SQL for regression tests with sql queries, but there are instances also where tests are also written in C like isolation tests, etc. How do we best judge which language is preferred to use when writing tests for Postgres components? How long should a test be when proposing them to hackers page?
4. In the src/test/examples directory (which are all libpq tests), why is the “examples” directory not included when building postgres? (Why weren't these libpq tests added to src/interface/libpq/test or in regression test suite instead?) In short, how to know where (in which file/directory in source code) to put a test?
On Wed, Nov 22, 2017 at 10:31 AM, Craig Ringer <craig@2ndquadrant.com> wrote: > On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison > <k.jamison@jp.fujitsu.com> wrote: >> I am entirely very new to PG development. >> >> Currently, I’m studying the test suites, and I feel the more experienced >> PG devs here could provide some insights on testing because I could not find >> concrete answers in the Postgres documentations. >> >> 1. How do you judge when a test suite is acceptable to be added to >> Postgres OSS source code? (How do you judge the value of a test suite?) > > Make your argument for it, and see if others agree. There's no formal > process. > > Inputs into the decision making process include: > > * How much coverage of previously untested functionality it adds > * How much code coverage it adds > * How long the test takes to run, especially considering the slow buildfarm > boxes and development turnaround time > * Whether the test fits into one of the existing suites we run routinely, or > requires separate steps > * How much work will be required to maintain the test Even with those factors in mind there is a case-by-case variability. Sometimes it is not worth initializing a new instance with initdb for a small test. Parallelization of tests help on large machines by on buildfarm members the size of a RPI things get time-consuming. Priority is usually given into things that do not consume much time, so as any developer can run the full set of tests quickly on a laptop. I have for example a small alias to run the full set of basic tests (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within two minutes if I recall correctly, TAP tests included. So runtime is really an important matter. >> 3. Is there a standard way of writing tests on the source code that >> we should follow, like when should test be written in TAP/SQL/C formats and >> how long should it be? >> >> ① I know that TAP test is for client program tests, SQL for >> regression tests with sql queries, but there are instances also where tests >> are also written in C like isolation tests, etc. How do we best judge which >> language is preferred to use when writing tests for Postgres components? How >> long should a test be when proposing them to hackers page? > > In general, prefer pg_regress if you can use it. Isolation tests for > concurrency issues. TAP tests if you can't write it with pg_regress or > isolation tester. Test modules only if you really must. This really depends on what you want to achieve, so this comes back to a case-by-case. Regression tests are fine for basic tests, where one session is enough to prove your feature. Isolation tests are helpful when testing concurrent behaviors across multple sessions. TAP tests are useful when more than one instance is needed or when you need to test cluster-wide configurations or client binaries. Having tests in patches proposed definitely helps people in understanding more how something works. Sometimes things get discarded but they can really help in making a point. Lately, while working on channel binding support for SCRAM, TAP tests have for example proved to help: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9288d62bb4b6f302bf13bb2fed3783b61385f315#patch13. This needs a SSL setup with a specific configuration for pg_hba.conf, and proved with just a lookup at the tests in the patch that the feature proposed was able to achieve what it aimed at. So it proved a point. >> 4. In the src/test/examples directory (which are all libpq tests), >> why is the “examples” directory not included when building postgres? (Why >> weren't these libpq tests added to src/interface/libpq/test or in regression >> test suite instead?) In short, how to know where (in which file/directory in >> source code) to put a test? > > Dunno, sorry. Those are mainly present as example programs. -- Michael
Craig Ringer <craig@2ndquadrant.com> writes:
> On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <
> k.jamison@jp.fujitsu.com> wrote:
>> 1. How do you judge when a test suite is acceptable to be added to
>> Postgres OSS source code? (How do you judge the value of a test suite?)
> Make your argument for it, and see if others agree. There's no formal
> process.
Yeah. Also realize that we've been accreting test cases for ~20 years,
and there's never been any master plan about what to test. So there's a
lot of individual test cases of varying quality/value, and you shouldn't
necessarily take any one existing test as gospel. I don't hold that out
as a great example of software engineering, but it's the truth.
> Inputs into the decision making process include:
> * How much coverage of previously untested functionality it adds
> * How much code coverage it adds
> * How long the test takes to run, especially considering the slow buildfarm
> boxes and development turnaround time
> * Whether the test fits into one of the existing suites we run routinely,
> or requires separate steps
> * How much work will be required to maintain the test
Also, portability/repeatability. If it doesn't produce the same answers,
with very high probability, across all the platforms we support, it isn't
going to last long.
>> 3. Is there a standard way of writing tests on the source code that
>> we should follow, like when should test be written in TAP/SQL/C formats and
>> how long should it be?
> In general, prefer pg_regress if you can use it. Isolation tests for
> concurrency issues. TAP tests if you can't write it with pg_regress or
> isolation tester. Test modules only if you really must.
Right. This ties in closely to the incremental runtime needed for one
additional test case. A new query or two in an existing SQL regression
script adds little overhead. A new TAP test adds quite a bit.
>> 4. In the src/test/examples directory (which are all libpq tests),
>> why is the “examples” directory not included when building postgres?
I've always taken those to be more documentation, or sample code, than
test cases. I'm not sure why they're under src/test/ at all.
regards, tom lane
>>>> 1. How do you judge when a test suite is acceptable to be added to >>>> Postgres OSS source code? (How do you judge the value of a test >>>> suite?) >>> >>> Make your argument for it, and see if others agree. There's no formal >>> process. > > Yeah. Also realize that we've been accreting test cases for ~20 years, > and there's never been any master plan about what to test. So there's a > lot of individual test cases of varying quality/value, and you shouldn't > necessarily take any one existing test as gospel. I don't hold that out > as a great example of software engineering, but it's the truth. >>> >>> Inputs into the decision making process include: >>> * How much coverage of previously untested functionality it adds >>> * How much code coverage it adds >>> * How long the test takes to run, especially considering the slow >>> buildfarm boxes and development turnaround time >>> * Whether the test fits into one of the existing suites we run >>> routinely, or requires separate steps >>> * How much work will be required to maintain the test >> >> Even with those factors in mind there is a case-by-case variability. >> Sometimes it is not worth initializing a new instance with initdb for >> a small test. Parallelization of tests help on large machines by on >> buildfarm members the size of a RPI things get time-consuming. >> Priority is usually given into things that do not consume much time, >> so as any developer can run the full set of tests quickly on a laptop. >> I have for example a small alias to run the full set of basic tests >> (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within >> two minutes if I recall correctly, TAP tests included. So runtime is >> really an important matter. > > Also, portability/repeatability. If it doesn't produce the same answers, > with very high probability, across all the platforms we support, it isn't > going to last long. Thank you everyone for your valuable input. I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not coveredby tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still learningthe PG internals and hacking. >>>> 3. Is there a standard way of writing tests on the source code that >>>> we should follow, like when should test be written in TAP/SQL/C >>>> formats and how long should it be? >>> >>> In general, prefer pg_regress if you can use it. Isolation tests for >>> concurrency issues. TAP tests if you can't write it with pg_regress >>> or isolation tester. Test modules only if you really must. > > Right. This ties in closely to the incremental runtime needed for one > additional test case. A new query or two in an existing SQL regression > script adds little overhead. A new TAP test adds quite a bit. I understand that tests should not take that long when running. For example, how about when proposing tests that just test parameter settings of a function? Let's say, for example, a libpq main function where in different test cases the params are null, empty, invalid, or valid. Are these test types common or recommended? >>>> 4. In the src/test/examples directory (which are all libpq tests), >>>> why is the “examples” directory not included when building postgres? >>> >>> Dunno, sorry. >> >> Those are mainly present as example programs. > > I've always taken those to be more documentation, or sample code, than > test cases. I'm not sure why they're under src/test/ at all. Since these example codes are only documentations, so should these be removed in the source code instead? Or should theseexamples code be revised and be moved to a test directory in libpq? Thanks again! Regards, Kirk Jamison
On Fri, Nov 24, 2017 at 10:27 AM, Jankirk.Vincent., Jamison <k.jamison@jp.fujitsu.com> wrote: > I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still notcovered by tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still learningthe PG internals and hacking. That's a good idea to get more familiar with the internals of Postgres. Don't forget to submit any patch you would like to get integrated into -hackers and to get it registered in the commit fest application. Please see here for more details: https://wiki.postgresql.org/wiki/Submitting_a_Patch -- Michael