Cover Python programs by soft&curly tests

There are several subconcepts of Code coverage, which is just a quantitative measure of finding out how much of the code has been executed. The concept can be deceptive, though, if one doesn't know exactly what those figures mean.

For example, all of the following are subconcepts of code coverage:

• Statement coverage
• Line coverage
• Condition coverage
• Decision coverage
• Multiple condition coverage
• Path coverage
• etc.

100% line coverage doesn't guarantee anything (well, except that the interpreter has traversed each line, nothing more :). For example, consider the line

if a.bar() and b.frob() or c.frob():
 do_something()

if a.bar() evaluates to false, b.frob() isn't usually executed at all due to ShortCircuit evaluation. b.frob() can be as broken as ever, but still line coverage can read 100% thus lulling developer into false sense of security. Condition coverage is a bit better, but it still doesn't try everything: what about loops? And even if a loop is executed no times at all and maximum amount of iterations, it still doesn't test everything.

Certainly one discovers quickly that to really test all execution paths, you have to test all possible paths, and even that leaves questions (what about if some loop is executed only max/2 iterations? what about all possible inputs?).

Still, even line coverage is not useless. For developer, it allows one to notice those parts of codes that haven't been even glanced at yet. In interpreted languages, those parts can contain even typos which usually lead to immediate crashes (calling a non-existing method, for example). Part coverage tests much, much more, but unfortunately, the amount of test paths rises exponentially in each decision. For a simple module with 20 if statements, each having 2 possible alternatives, there are 2²⁰ ~ 1 million different paths of execution. Add to that a few loops and one more complex condition there and another elsewhere, and you quickly run into billions of test paths. In any real-life project, even 10% path coverage can be just impossible to achieve.

pip install coverage
pip freeze | grep coverage
# pip install --upgrade pip

Requirement already satisfied: coverage in /usr/local/share/DVCS/lib/Python/venv/lib/python3.4/site-packages
coverage==4.3.4