I’m in agreement with Robert Glass when he says “100% test coverage is insufficient. 35% of the faults are missing logic paths.” It’s not controversial, but I’d like to give my perspective on it.
If you have an automated unit test suite, low code coverage is an indication that you need more tests. Unfortunately, high code coverage does not tell you if you have enough tests or the right tests. Adding to Robert Glass’ observation, executed code is not necessarily tested code. Imagine a test case that runs through many lines of code, but never checks that they are doing the right thing. At best this is the “I don’t have any bad pointers” test.
Here are Gerard Meszaros’ steps of the Four Phase test pattern
- Setup preconditions
- Exercise the code under test
- Check the desired behavior
- Restore the system to its original state
You can get 100% test coverage (executing every line of code) and never do step 3. The code is initialized and exercised, but we are not sure if it is correct. Low test coverage is telling, while high test coverage can be meaningless. Here is an example test case:
TEST(MyComplexThing, ScenarioA) { // init for scenario A result = MyComplexOperation(a,b,c,d); CHECK(result == TRUE); } TEST(MyComplexThing, ScenarioB) { // init for scenario B result = MyComplexOperation(a,b,c,d); CHECK(result == TRUE); } TEST(MyComplexThing, ScenarioC) { // init for scenario C result = MyComplexOperation(a,b,c,d); CHECK(result == FALSE); } |
If MyComplexOperation() operation has a cyclomatic complexity of 3, I should be able to test it with three test cases, one for each execution scenario. If all MyComplexOperation() does is return a result, this is adequate. If it interacts with some other part of the system, it is not adequate. There are no checks to see if it did the right thing.
Metrics are important, and it is equally important to understand what they do and do not tell us.
The single most important thing considering quality is a team that cares about what it builds.
Code coverage as a goal is just worthless, however I often use it as an indicator. Especially when refactoring code to check that nothing becomes untested. Because I strictly work with ATDD and TDD, building new stuff is covered anyway.
Thanks for the post
Urs
100% line coverage says nothing, like you said. Rather try to cover more branches. This will probably force you to lookmore at behavior. 🙂 (i do agree that behavior is what matters)
A little note about 100% source code coverage in C++: always a 100% coverage don’t means that you will test all your source code so I will assume we wrote the software in C.
The problem is not (only) to have 100% source code coverage, the problem is to have 100% requirements coverage.
So before we should agree to have a set of executable (acceptance) tests and we should agree they cover requirements at 100% if they are all passed.
Then we can speak about source code coverage. If our application meets at 100% the (executable) requirements and the coverage is less then 100% we should had written some source code that does nothing useful or we made some wrong assumptions or we miss some hidden requirements for what we didn’t provide a test.
For these reason I prefer a little bit BDD then TDD to develop software.
I think 100% coverage is useful as a goal. But if you treat it as the metric that once met ends all testing development or effort, you are sadly mistaken. That’s like assuming my team will win a basketball game if I can hit 3 point shots 100% of the time. While hitting 100% of all 3 pointers is amazing and will definitely improve team performance, there’s more to basketball than hitting shots. For example, if we never worked on passing me the ball, my 3 point ability becomes useless.
I agree @Iheany. Not having close to 100% tells you a lot, though having 100% says nothing of the quality of the test cases at preserving the desired behavior of code.