Unit testing is a tool, not a goal

Tom Moertel — Tue, 10 Oct 2006 15:23:00 -0400

Via the ever-entertaining programming.reddit.com, I discovered The (Misguided?) Faith in Unit Tests. I don’t agree with the article’s thesis, but the article did hit upon a grain of unfortunate truth: many unit-testing practitioners have a rather cultish aspect to their devotion. More and more, people push unit testing not because it’s an inexpensive way to establish a lot of confidence in our code’s correctness, but because it’s “good” or “a best practice” or “professional.”

When unit testing becomes an end in itself, something has gone haywire. Software-development practices, including unit testing, are just tools. No one tool is the solution to all problems. To the extent that we over-apply any particular tool, we cheat ourselves. We miss out on the opportunity to use other, more effective tools.

My favorite example of a common, important kind of problem that unit testing isn’t much help for is the “strings problem.” All those cross-site-scripting vulnerabilities and SQL-injection holes you have been reading about in recent days are prime examples. In short, the strings problem is having to keep track of and escape, as needed, the bazillion types of strings that flow as life-blood within the arteries of modern web applications.

Trying to detect string-escaping problems via unit testing is costly because it’s at least as difficult to write the tests correctly as it is to write the code the tests are testing. Further, a programmer who forgets that he needs to escape user-supplied strings before stuffing them into live web pages isn’t going to remember to write tests for his having forgotten to escape the strings.

If, then, you come to the programming party with the preconceived notion that unit testing is the only way to go, there’s a pretty good chance that your web applications are going to be sporting some XSS or SQL-injection holes. On the other hand, if you view unit testing as a tool, you’re likely to consider the possibility that other tools might be more effective for something like the strings problem. You might, for example, decide to use a type-system-based solution, which pretty much makes the strings problem trivial to solve. (I’ll write about solving the strings problem in Haskell in an upcoming article.)

I’m not saying that unit testing is “bad” or that it’s not “good.” What I’m saying is that unit testing is not the goal. The goal is having confidence in your code doing what it ought to do. Unit testing is often an inexpensive way to achieve much of that confidence, but for many situations, the strings problem being one of them, it pays to keep an open mind about supplementing unit testing with other tools.

Update: Minor edits for readability.

Tom Moertel's Weblog: Unit testing is a tool, not a goal

Unit testing is a tool, not a goal