Thread: On Judging the Value of Tests

On Judging the Value of Tests

From
"Jankirk.Vincent., Jamison"
Date:

Dear PG Experts,

 

I am entirely very new to PG development.

Currently, I’m studying the test suites, and I feel the more experienced PG devs here could provide some insights on testing because I could not find concrete answers in the Postgres documentations.

 

1.      How do you judge when a test suite is acceptable to be added to Postgres OSS source code? (How do you judge the value of a test suite?)

 

2.      Besides code coverage test, are there other ways/techniques/tools to measure how much value a test would have?

This is coming from a perspective that in the current code coverage (https://coverage.postgresql.org/), some functions were covered because the function was called/run in some tests, while the function/lines may also not necessarily be thoroughly/directly tested so the coverage skips some lines of code inside a block.

 

3.      Is there a standard way of writing tests on the source code that we should follow, like when should test be written in TAP/SQL/C formats and how long should it be?

      I know that TAP test is for client program tests, SQL for regression tests with sql queries, but there are instances also where tests are also written in C like isolation tests, etc. How do we best judge which language is preferred to use when writing tests for Postgres components? How long should a test be when proposing them to hackers page?

             

4.      In the src/test/examples directory (which are all libpq tests), why is the “examples” directory not included when building postgres? (Why weren't these libpq tests added to src/interface/libpq/test or in regression test suite instead?) In short, how to know where (in which file/directory in source code) to put a test?

 

I’d like to apologize for the series of questions, but I’d really like to get insights from the experienced devs.

It's alright if you don't have answers to all questions as this might take much of your time.

Thank you very much.

 

 

Regards,

Kirk Jamison

Re: On Judging the Value of Tests

From
Craig Ringer
Date:
On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <k.jamison@jp.fujitsu.com> wrote:

Dear PG Experts,

 

I am entirely very new to PG development.

Currently, I’m studying the test suites, and I feel the more experienced PG devs here could provide some insights on testing because I could not find concrete answers in the Postgres documentations.

 

1.      How do you judge when a test suite is acceptable to be added to Postgres OSS source code? (How do you judge the value of a test suite?)


Make your argument for it, and see if others agree. There's no formal process.

Inputs into the decision making process include:

* How much coverage of previously untested functionality it adds
* How much code coverage it adds
* How long the test takes to run, especially considering the slow buildfarm boxes and development turnaround time
* Whether the test fits into one of the existing suites we run routinely, or requires separate steps
* How much work will be required to maintain the test

3.      Is there a standard way of writing tests on the source code that we should follow, like when should test be written in TAP/SQL/C formats and how long should it be?

      I know that TAP test is for client program tests, SQL for regression tests with sql queries, but there are instances also where tests are also written in C like isolation tests, etc. How do we best judge which language is preferred to use when writing tests for Postgres components? How long should a test be when proposing them to hackers page?


In general, prefer pg_regress if you can use it. Isolation tests for concurrency issues.  TAP tests if you can't write it with pg_regress or isolation tester. Test modules only if you really must.

             

4.      In the src/test/examples directory (which are all libpq tests), why is the “examples” directory not included when building postgres? (Why weren't these libpq tests added to src/interface/libpq/test or in regression test suite instead?) In short, how to know where (in which file/directory in source code) to put a test? 

Dunno, sorry.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: On Judging the Value of Tests

From
Michael Paquier
Date:
On Wed, Nov 22, 2017 at 10:31 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison
> <k.jamison@jp.fujitsu.com> wrote:
>> I am entirely very new to PG development.
>>
>> Currently, I’m studying the test suites, and I feel the more experienced
>> PG devs here could provide some insights on testing because I could not find
>> concrete answers in the Postgres documentations.
>>
>> 1.      How do you judge when a test suite is acceptable to be added to
>> Postgres OSS source code? (How do you judge the value of a test suite?)
>
> Make your argument for it, and see if others agree. There's no formal
> process.
>
> Inputs into the decision making process include:
>
> * How much coverage of previously untested functionality it adds
> * How much code coverage it adds
> * How long the test takes to run, especially considering the slow buildfarm
> boxes and development turnaround time
> * Whether the test fits into one of the existing suites we run routinely, or
> requires separate steps
> * How much work will be required to maintain the test

Even with those factors in mind there is a case-by-case variability.
Sometimes it is not worth initializing a new instance with initdb for
a small test. Parallelization of tests help on large machines by on
buildfarm members the size of a RPI things get time-consuming.
Priority is usually given into things that do not consume much time,
so as any developer can run the full set of tests quickly on a laptop.
I have for example a small alias to run the full set of basic tests
(make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within
two minutes if I recall correctly, TAP tests included. So runtime is
really an important matter.

>> 3.      Is there a standard way of writing tests on the source code that
>> we should follow, like when should test be written in TAP/SQL/C formats and
>> how long should it be?
>>
>> ①      I know that TAP test is for client program tests, SQL for
>> regression tests with sql queries, but there are instances also where tests
>> are also written in C like isolation tests, etc. How do we best judge which
>> language is preferred to use when writing tests for Postgres components? How
>> long should a test be when proposing them to hackers page?
>
> In general, prefer pg_regress if you can use it. Isolation tests for
> concurrency issues.  TAP tests if you can't write it with pg_regress or
> isolation tester. Test modules only if you really must.

This really depends on what you want to achieve, so this comes back to
a case-by-case. Regression tests are fine for basic tests, where one
session is enough to prove your feature. Isolation tests are helpful
when testing concurrent behaviors across multple sessions. TAP tests
are useful when more than one instance is needed or when you need to
test cluster-wide configurations or client binaries. Having tests in
patches proposed definitely helps people in understanding more how
something works. Sometimes things get discarded but they can really
help in making a point. Lately, while working on channel binding
support for SCRAM, TAP tests have for example proved to help:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9288d62bb4b6f302bf13bb2fed3783b61385f315#patch13.
This needs a SSL setup with a specific configuration for pg_hba.conf,
and proved with just a lookup at the tests in the patch that the
feature proposed was able to achieve what it aimed at. So it proved a
point.

>> 4.      In the src/test/examples directory (which are all libpq tests),
>> why is the “examples” directory not included when building postgres? (Why
>> weren't these libpq tests added to src/interface/libpq/test or in regression
>> test suite instead?) In short, how to know where (in which file/directory in
>> source code) to put a test?
>
> Dunno, sorry.

Those are mainly present as example programs.
--
Michael


Re: On Judging the Value of Tests

From
Tom Lane
Date:
Craig Ringer <craig@2ndquadrant.com> writes:
> On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <
> k.jamison@jp.fujitsu.com> wrote:
>> 1.      How do you judge when a test suite is acceptable to be added to
>> Postgres OSS source code? (How do you judge the value of a test suite?)

> Make your argument for it, and see if others agree. There's no formal
> process.

Yeah.  Also realize that we've been accreting test cases for ~20 years,
and there's never been any master plan about what to test.  So there's a
lot of individual test cases of varying quality/value, and you shouldn't
necessarily take any one existing test as gospel.  I don't hold that out
as a great example of software engineering, but it's the truth.

> Inputs into the decision making process include:

> * How much coverage of previously untested functionality it adds
> * How much code coverage it adds
> * How long the test takes to run, especially considering the slow buildfarm
> boxes and development turnaround time
> * Whether the test fits into one of the existing suites we run routinely,
> or requires separate steps
> * How much work will be required to maintain the test

Also, portability/repeatability.  If it doesn't produce the same answers,
with very high probability, across all the platforms we support, it isn't
going to last long.

>> 3.      Is there a standard way of writing tests on the source code that
>> we should follow, like when should test be written in TAP/SQL/C formats and
>> how long should it be?

> In general, prefer pg_regress if you can use it. Isolation tests for
> concurrency issues.  TAP tests if you can't write it with pg_regress or
> isolation tester. Test modules only if you really must.

Right.  This ties in closely to the incremental runtime needed for one
additional test case.  A new query or two in an existing SQL regression
script adds little overhead.  A new TAP test adds quite a bit.

>> 4.      In the src/test/examples directory (which are all libpq tests),
>> why is the “examples” directory not included when building postgres?

I've always taken those to be more documentation, or sample code, than
test cases.  I'm not sure why they're under src/test/ at all.
        regards, tom lane


RE: On Judging the Value of Tests

From
"Jankirk.Vincent., Jamison"
Date:
>>>> 1.      How do you judge when a test suite is acceptable to be added to
>>>> Postgres OSS source code? (How do you judge the value of a test 
>>>> suite?)
>>>
>>> Make your argument for it, and see if others agree. There's no formal 
>>> process.
>
> Yeah.  Also realize that we've been accreting test cases for ~20 years,
> and there's never been any master plan about what to test.  So there's a 
> lot of individual test cases of varying quality/value, and you shouldn't
> necessarily take any one existing test as gospel. I don't hold that out 
> as a great example of software engineering, but it's the truth.
>>>
>>> Inputs into the decision making process include:
>>> * How much coverage of previously untested functionality it adds
>>> * How much code coverage it adds
>>> * How long the test takes to run, especially considering the slow 
>>> buildfarm boxes and development turnaround time
>>> * Whether the test fits into one of the existing suites we run 
>>> routinely, or requires separate steps
>>> * How much work will be required to maintain the test
>>
>> Even with those factors in mind there is a case-by-case variability.
>> Sometimes it is not worth initializing a new instance with initdb for
>> a small test. Parallelization of tests help on large machines by on 
>> buildfarm members the size of a RPI things get time-consuming.
>> Priority is usually given into things that do not consume much time, 
>> so as any developer can run the full set of tests quickly on a laptop.
>> I have for example a small alias to run the full set of basic tests 
>> (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within 
>> two minutes if I recall correctly, TAP tests included. So runtime is 
>> really an important matter.
>
> Also, portability/repeatability.  If it doesn't produce the same answers, 
> with very high probability, across all the platforms we support, it isn't 
> going to last long.

Thank you everyone for your valuable input.
I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not
coveredby tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still
learningthe PG internals and hacking.
 

>>>> 3.      Is there a standard way of writing tests on the source code that
>>>> we should follow, like when should test be written in TAP/SQL/C 
>>>> formats and how long should it be?
>>>
>>> In general, prefer pg_regress if you can use it. Isolation tests for 
>>> concurrency issues.  TAP tests if you can't write it with pg_regress 
>>> or isolation tester. Test modules only if you really must.
>
> Right.  This ties in closely to the incremental runtime needed for one 
> additional test case.  A new query or two in an existing SQL regression 
> script adds little overhead.  A new TAP test adds quite a bit.

I understand that tests should not take that long when running.
For example, how about when proposing tests that just test
parameter settings of a function? 
Let's say, for example, a libpq main function where in different test cases 
the params are null, empty, invalid, or valid.
Are these test types common or recommended?

>>>> 4.      In the src/test/examples directory (which are all libpq tests),
>>>> why is the “examples” directory not included when building postgres?
>>>
>>> Dunno, sorry.
>>
>> Those are mainly present as example programs.
>
> I've always taken those to be more documentation, or sample code, than
> test cases.  I'm not sure why they're under src/test/ at all.

Since these example codes are only documentations, so should these be removed in the source code instead? Or should
theseexamples code be revised and be moved to a test directory in libpq?
 

Thanks again!

Regards,
Kirk Jamison

Re: On Judging the Value of Tests

From
Michael Paquier
Date:
On Fri, Nov 24, 2017 at 10:27 AM, Jankirk.Vincent., Jamison
<k.jamison@jp.fujitsu.com> wrote:
> I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still
notcovered by tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still
learningthe PG internals and hacking. 

That's a good idea to get more familiar with the internals of
Postgres. Don't forget to submit any patch you would like to get
integrated into -hackers and to get it registered in the commit fest
application. Please see here for more details:
https://wiki.postgresql.org/wiki/Submitting_a_Patch
--
Michael