Tom Moertel’s Blog

Home About Archive

In software, as in life, every problem you solve is actually part of a larger problem, an optimization problem in which you are trying to squeeze as much as you can from a single scarce resource: yourself. Every hour you spend solving one problem is an hour you no longer have to solve other problems. At the end of your life, when you receive the final tally of how you spent your scant allotment of hours, you probably don’t want to discover that you wasted most of them on problems that didn’t matter.

Which brings us to David Heinemeier Hansson’s recent post on Testing like the TSA, in which he advises against wasting your time writing tests that are unlikely to find problems. He’s right.

Whether you realize it or not, every time you write a test, you are placing a bet. You are betting that the test’s expected payoff will outweigh the test’s expected opportunity cost. Instead of writing that test, would you be better off doing something else? For every test you write, you are betting that the answer is no.

But being right about this answer – reliably right – is tricky. It requires that you have a good understanding of which bets are likely to pay off and which are not, which are cheap to place and which are expensive. In other words, it requires that your internal probability distributions be well calibrated for the kind of code you write and the environment in which you code.

New programmers, unfortunately, are poorly calibrated. They have little experience to draw upon and end up imitating other people’s probability distributions. These they pick up, for better or worse, from folklore – best practices, development methods, expert opinion, the guy down the hall. As a result, new programmers concentrate their nascent probability distributions around the things they learn first and, consequently, over-bet on those things.

But even the best programmers over-bet on the things they know. That’s because they under-bet in their blind spots, and we all have blind spots.

In general, we never bet on the things we don’t know. If all you know about getting your code right is TDD, you’ll never bet on types or proofs or constructive correctness because you don’t know how to place those bets. But those bets are often dirt cheap and pay in spades. If you’re not betting on them at least some of the time, whatever you are betting on probably costs more and pays less. You could be doing better.

But to do better, you have to know what other bets are out there. You have to know what it costs to place them and how likely they are to pay off. To gain this knowledge, you’ll have to go beyond what is comfortable; you’ll have to explore those dark, distant parts of the casino and place some trial bets.

If you’ve never written a unit test, write some. If you always write your tests after your main logic, write them before. If you always write them before, write them after. If all you ever do is unit testing, try integration and system tests, too. If you’ve never done QuickCheck-like property checking, do it. If you don’t know how to prove things about your code, write some proofs. If you don’t know how to represent important properties as types so that your computer will prove them for you, now’s the time to learn. If you don’t know how to design APIs so that certain classes of problems become impossible, start practicing.

And, after you place your bets, pay attention to what happens down the road. Learn from what you observe. Learn which bets are likely pay off now, pay off later, and pay off never.

The more bets you know how to place, and the more you know about their expected payoffs, the better you’ll place your bets. And the better you place your bets, the more problems you’ll solve, problems that actually matter, before you run of out of time.

Remember: Your supply of hours is finite. If you care about making those hours matter, learn how to bet them wisely. Both in software, and in life.