Full Test Coverage is Impractical
Many developers claim that to achieve high quality software, developers must create automated tests that ensure that all possible execution routes have been covered. This is also known as full path coverage. I will argue that different types of software require different testing approaches and that full path coverage is impractical in almost every case. Too many tests simply create clutter.
Let’s look at the impracticality first. Writing tests requires skill and effort. Everybody on your team is probably not a testing expert. Not only may it be difficult to get started with testing, but you will generate a great amount of code with its load of maintenance. Tests are not immune to their own errors either. I have uncovered a number of incorrect test cases in every project, including PHP itself.
A full coverage can give a fake sense of security, because you won’t second guess your tests and there will be too many to effectively audit them. That doesn’t mean that more tests is bad. It means that you must strike a good balance between the number of tests and the assurance that you are seeking. Look at it as a car insurance that costs more per year than the price on your car. You want to feel safe with the right insurance, but at some point, it’s more trouble than it’s worth. You can easily end up with more test code than software code, so watch out.
My second point is that different software justifies different tests. Software that is meant to be used by other software — such as a programming language, a framework or an API — would require more in-depth testing. I would recommend writing a few tests per function, to make sure that they return expected values. Testing all paths may still not be necessary, because some paths may prove redundant. Let’s take a date parsing function as an example. If you can parse “2013-05-22 13:00″ and “May 22, 2013 1:00pm” correctly, then perhaps testing the permutation “2013-05-22 1:00pm” may be unnecessary. Be smart about your tests: choose quality over quantity.
For consumer software, you may not need to test everything. I recommend to skip obvious tests. For example, it is obvious that when you first instanciate this class, getTotal() will return zero.
class ShoppingCart {
protected $total = 0;
function getTotal() {
return $this->total;
}
}
Writing overly obvious test cases is like underlining text that is already highlighted. I usually start testing areas that can have impact the business, such as losing data or selling 0.01 quantity of a product.
In the end, there are no exact rules on how much to test. The most important thing to keep in mind is that writing tests for the sake writing tests is futile and costly. Not only that, but your colleagues won’t bother reading or maintaining tests if they are only there for their own sake and provide too little value. Focus on building great software. Tests are a tool to make it better. Just don’t overdo it.
I often feel I want to teach developers about the business side of projects. That a project is an investment that may or may not pay off. And therefore that technical debt goes hand in hand with that – it’s better to have some debt you only have to pay if you are successful, than to pay it off right away even if you are not successful. If you’re successful, a business will automatically be generating money, so even if something costs a bit more, in a sense it still costs less. Plus there’s the issues of rapid prototyping and time to market. I don’t dare bring these things up with most developers though, because they are (righly or wrongly – it’s an interesting debate) very focused on solely their own job/role, making great non-fragile code. I often feel these kinds of pragmatic views are enough to get one burnt as a witch.
This said, most of the time I am trying to raise quality higher and higher in my company’s project. Most developers you will work with need to spend more time on quality, not less, especially for a long-serving code base. But the majority of speakers and writers, I think they’re on the other end. You’ve got to have a happy medium IMO . But as Anna’s article discusses, it’s all about context. I have an issue with the idea of “software craftsmanship” as an absolute. Yet, at the same time I can also agree with it – a developer isn’t a business person, so simply embracing their role and doing it well, is a great thing. My only real issue is when they bitch and moan and evangelise in inflexible and absolute terms.
Is the example a brain teaser?
I think, to clarify, that you’re getting at the idea of preventing detection overlapping. If the class is initialised with total as zero, and we add 1000, we expect the result to be 1000. There should be a test for this so problems with the initial value are detected, ergo a test for the initial value is redundant. In fact, your test suite should be screaming at you. That said, if there is a context where the initial value can change and not be detected by tests (e.g. scenarios where it’s not touched but will be queried by a client class), then it deserves its own test.
This is also my way of saying that good testing requires experience. If a developer lacks that experience, I’d strongly prefer they err on the side of caution (and balance bedamned). The results of inexperience on a test suite can be dramatic to the point of the test suite being fragile, highly aggregated (mistaking functional tests for unit tests) and written specifically to meet a min. coverage level (i.e. the tests are written to meet >80% coverage which doesn’t mean all necessary logic paths were exercised). You can see this even in “older” open source projects like the original Zend Framework unit tests. It’s also why I advise people to try and keep their assertion rate to 1 per test – it discourages some particularly bad practices.
It’s a bit simpler:
too many unit test = too high maintenance costs for a company
A rule of 80:20 should apply for all unit tests – means 80% of code coverage should ensure productive operativeness of the application. Remaining 20% will always be at risk, but costly effective.
You can’t generalize so much. Some software types can tolerate more risk than others. You can’t afford 20% of your code to remain untested when you send an expensive probe to Mars, when you hook a patient on life support or when you monitor a nuclear reactor. On the other hand, you don’t need to cover as much as 80% of your code when you build a social farming game, an online catalog for shoes or or a barcode interface for non-critical purposes.
The variables must be thoroughly understood and properly weighed. If you apply an arbitrary rule to all your projects, you can end up in a dangerous situation.
Graham It can be challenging to bring up business needs with developer, but the good developers will understand them when they are presented in the right way. After all a developer who doesn’t understand business value will produce something really cool that no one else uses, while a developer who gets it will focus their work so they can build an application that makes life easier for thousands or millions of people.
I disagree in regards to obvious tests, those ones are the low hanging fruit, you write them now without the need to put any thought into it. But in half a year or a year, when this test suddenly fails, you’re glad you wrote it, as it’s the base functionality that’s suddenly broken.
The rest makes sense, you don’t need to cover all cases (it would be nice, though), just add tests as soon as you encounter a bug.
They’re easy to write but typically have no value at all. Checking that a new array contains zero items is paranoid. Maintaining such tests would be like tending to weeds that prevent any real fruit to grow. It can be a way to learn testing, I suppose, but you must quickly learn to chase the valuable ones.