Thread: Coverage report

Hi,

Pursuant to promises made in Brussels a couple of months ago, I set up a
machine to run and expose the "make coverage" report under "make
check-world".  Some people have already heard about this.

I would like to collect ideas on how to improve this.  For example

* Should we run something other than "make check-world"  As far as I
know, that covers all or almost all the tests we have; are there things
that we should have and are not running?  If so, how do we go about
enabling them?

* buildfarm doesn't run make check-world for reasons of its own.  Maybe
we should run exactly what a typical buildfarm animal runs?  That way,
the coverage report would be closer to what we're actually verifying.

* Should we keep historical reports to study how numbers change in time?
Maybe save one report a week or something like that?

* Any other comments?

-- 
Álvaro Herrera                            http://www.linkedin.com/in/alvherre

Hello, Alvaro.

> Some people have already heard about this.

Yes, we did :) Nice job!

> * Should we run something other than "make check-world"  As far as I
> know, that covers all or almost all the tests we have; are there
> things that we should have and are not running?  If so, how do we go
> about enabling them?

I'm personally quite concerned that currently everything that is in .h
files is considered a public API and thus should not be changed, or some
third-party extensions can break. Can we run `make check` on some (or
maybe all?) extensions from PGXN? This way we could guarantee that if
changing API breaks some extensions, this are extensions that actually
nobody uses or knows about.

Not mentioning that in this case we will check code coverage of these
extensions too.

> * buildfarm doesn't run make check-world for reasons of its own.
> Maybe we should run exactly what a typical buildfarm animal runs?
> That way, the coverage report would be closer to what we're actually
> verifying.

How about something like `make check; make -C contrib check` ?

> * Should we keep historical reports to study how numbers change in
> time? Maybe save one report a week or something like that?

Yes, I think we should. One report a day and a history for a week or two
should be enough.

> * Any other comments?

I noticed that output of genhtml looks like this:

```
Writing directory view page.
Overall coverage rate: lines......: 63.2% (179603 of 284239 lines) functions..: 69.5% (10550 of 15183 functions)
```

Maybe we should send an e-mail to pgsql-hackers@ automatically if these
numbers drops.

-- 
Best regards,
Aleksander Alekseev
http://eax.me/