Re: Test code is worth the space - Mailing list pgsql-hackers
From | Jim Nasby |
---|---|
Subject | Re: Test code is worth the space |
Date | |
Msg-id | 55D23048.4050100@BlueTreble.com Whole thread Raw |
In response to | Re: Test code is worth the space (Greg Stark <stark@mit.edu>) |
Responses |
Re: Test code is worth the space
(Noah Misch <noah@leadboat.com>)
|
List | pgsql-hackers |
On 8/16/15 8:48 AM, Greg Stark wrote: > On Sun, Aug 16, 2015 at 7:33 AM, Noah Misch <noah@leadboat.com> wrote: >> When I've just spent awhile implementing a behavior change, the test diffs are >> a comforting sight. They confirm that the test suite exercises the topic I >> just changed. Furthermore, most tests today do not qualify under this >> stringent metric you suggest. The nature of pg_regress opposes it. > > It's not a comforting sight for me. It means I need to change the test > results and then of course the tests will pass. So they didn't really > test anything and I don't really know whether the changes broke > anything. Any test I had to adjust for my change was effectively > worthless. > > This is precisely my point about pg_regress and why it's the reason > there's pressure not to have extensive tests. It's useful to have some > end-to-end black box tests like this but it's self-limiting. You can't > test many details without tying the tests to specific behaviour. > > I have worked with insanely extensive testing suites that didn't have > this problem. But they were unit tests for code that was structured > around testability. When the API is designed to be testable and you > have good infrastructure for mocking up the environment and comparing > function results in a much narrower way than just looking at text > output you can test the correctness of each function without tying the > test to the specific behaviour. > > *That* gives a real comfort. When you change a function to behave > entirely differently and can still run all the tests and see that all > the code that depends on it still operate correctly then you know you > haven't broken anything. In that case it actually *speeds up* > development rather than slowing it down. It lets newbie developers > hack on code where they don't necessarily know all the downstream > dependent code and not be paralyzed by fear that they'll break > something. As someone who oversaw a database test suite with almost 500 test files (each testing multiple cases), I completely agree. Our early framework was actually similar to pg_regress and that worked "OK" up to about 200 test files, but it got very unwieldy very quickly. We switched to pgTap and it was far easier to work with. I suspect any effort to significantly improve Postgres test coverage is doomed until there's an alternative to pg_regress. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Data in Trouble? Get it in Treble! http://BlueTreble.com
pgsql-hackers by date: