There are several subconcepts of Code coverage, which is just a quantitative measure of finding out how much of the code has been executed. The concept can be deceptive, though, if one doesn't know exactly what those figures mean.

For example, all of the following are subconcepts of code coverage:

  • Statement coverage
  • Line coverage
  • Condition coverage
  • Decision coverage
  • Multiple condition coverage
  • Path coverage
  • etc.


100% line coverage doesn't guarantee anything (well, except that the interpreter has traversed each line, nothing more :). For example, consider the line

if and b.frob() or c.frob():

if evaluates to false, b.frob() isn't usually executed at all due to ShortCircuit evaluation. b.frob() can be as broken as ever, but still line coverage can read 100% thus lulling developer into false sense of security. Condition coverage is a bit better, but it still doesn't try everything: what about loops? And even if a loop is executed no times at all and maximum amount of iterations, it still doesn't test everything.

Problem scale

Certainly one discovers quickly that to really test all execution paths, you have to test all possible paths, and even that leaves questions (what about if some loop is executed only max/2 iterations? what about all possible inputs?).

Still, even line coverage is not useless. For developer, it allows one to notice those parts of codes that haven't been even glanced at yet. In interpreted languages, those parts can contain even typos which usually lead to immediate crashes (calling a non-existing method, for example). Part coverage tests much, much more, but unfortunately, the amount of test paths rises exponentially in each decision. For a simple module with 20 if statements, each having 2 possible alternatives, there are 220 ~ 1 million different paths of execution. Add to that a few loops and one more complex condition there and another elsewhere, and you quickly run into billions of test paths. In any real-life project, even 10% path coverage can be just impossible to achieve.

Pertinent solution

Still, path coverage may be useful for testing only single methods which contain complex algorithms. For Python, it seems like we have only statement coverage so far, thanks to efforts of Gareth Rees and Ned Batchelder. More advanced tools are under development, however.


- Code coverage measurement for Python by Ned Batchelder

Code coverage


pip install coverage
pip freeze | grep coverage
# pip install --upgrade pip
Requirement already satisfied: coverage in /usr/local/share/DVCS/lib/Python/venv/lib/python3.4/site-packages

Test launch

Single Python

Python in suite

Python in elegant suite

blog comments powered by Disqus


22 December 2016


literate programming Python coverage