Tom Moertel's Weblog: Tag testing

Property checking with Python's nose testing framework

Tom Moertel — Wed, 19 Mar 2008 22:34:00 -0400

At work recently I was writing some tests with Python’s out-of-the-box unit-testing framework unittest. I’m new to Python and accustomed to Perl and Haskell’s testing frameworks, which are lightweight and let you write tests without much hoop-jumping. In particular, QuickCheck and LectroTest make it easy to test at the property level instead of the test-case level. With unittest, I was having to write a lot of code to get the same level of abstraction.

By “property level,” here’s what I mean. Say I’m testing this thing, let’s call it a subscriber pool. It has two fundamental properties:

Subscribe. For all initial states of the pool, if you call subscribe(user), then, assuming there have been no other operations on the pool, user must be in the pool.
Unsubscribe. For all initial states of the pool, if you call unsubscribe(user), then, assuming there have been no other operations on the pool, user must not be in the pool.

That’s it. If my implementation satisfies both properties, it’s correct. (This is a simplified version of my real testing problem, which required additional property checks.)

To test whether my implementation satisfies each property, I must write individual test cases that together “cover” the property. For example, to test whether the Subscribe property holds, I might write four test cases:

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testEmpty(self):
        load_pool_with_members([])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testOtherGuyAlreadyInPool(self):
        load_pool_with_members(["2"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAlreadyInPool(self):
        load_pool_with_members(["1"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAndOtherGuyAlreadyInPool(self):
        load_pool_with_members(["1", "2"])
        subscribe("1")
        self.assert_("1" in pool_members())

Every one of the test cases has the same form. The repetition makes me want to refactor the whole thing.

Okay, let’s do it:

INITIAL_POOL_STATES = [[], ["2"], ["1"], ["1", "2"]]

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

We’re fighting a bit with the testing framework because our notion of when set-up and tear-down should occur doesn’t match its own, but otherwise our code is looking much more manageable. In particular, if we want to extend our property-check coverage with additional initial pool states, we don’t need to write additional tests; instead, we can just extend a single list.

But we’re only halfway done. We must also check the Unsubscribe property. The code for it is virtually the same as for Subscribe, but with subscribe becoming unsubscribe and in becoming not in. Let’s add it to our class:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

    def testUnsubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                unsubscribe("1")
                self.assert_("1" not in pool_members(), case)
            finally:
                self.tearDown()

And now let’s factor out the new redundancy:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        def testfn(case):
            subscribe("1")
            self.assert_("1" in pool_members(), case)
        self._forall_test_cases(testfn)

    def testUnsubscribe(self):
        def testfn(case):
            unsubscribe("1")
            self.assert_("1" not in pool_members(), case)
        self._forall_test_cases(testfn)

    def _forall_test_cases(self, testfn):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                testfn(case)
            finally:
                self.tearDown()

It’s not bad, but it’s not great either. There’s still a lot of noise in that code.

After discussing the situation with my more-Pythonic colleague Tim Lesher, I took his advice to check out the nose testing framework.

One of the things I liked right away about nose was that it supports test generators, which would let me represent each property-check as a generator that yields the test cases needed to check the property. Also, set-up and tear-down would automatically occur per generated test, so I wouldn’t have to invoke them manually.

Once I got familiar with nose, it was easy to create a decorator to represent the forall-test-cases idiom:

def forall_cases(cases):
    def decorate(testfn):
        def gen():
            for case in cases:
                yield testfn, case
        gen.__name__ = "test_%s_for_a_case" % testfn.__name__
        return gen
    return decorate

Note that this decorator is not specific to our subscriber-pool tests. It can be used in any situation where we need to check a property across a collection of cases. In fact, I keep this little gem in a “nosehelpers” library, where I reuse it all the time. Here’s an example of how to use it to check the trivial property that x = x for all x in 0–99:

@forall_cases(range(100))
def check_self_equality(x):
    assert x == x

Now, back to our testing problem. Here’s how we can use the decorator to check the Subscribe property:

@forall_cases(INITIAL_POOL_STATES)
@with_setup(initialize_pool, destroy_pool)
def check_subscribe(case):
    load_pool_with_members(case)
    subscribe("1")
    assert "1" in pool_members()

(The with_setup decorator is defined by nose and tells nose to run the given set-up and tear-down functions before and after each of the generated test cases.)

Not bad. The only problem I have with that code is that it mixes the “For all initial states of the pool” part of the property definition into the “if you call subscribe(user), then …” part. I’d like the code to be more explicit about which part defines the scope of the property claim and which part defines the test for whether the claim holds for any particular test case within that scope.

Fortunately, we can build upon our existing decorator to create exactly what we need:

def forall_initial_pools(testfn):
    @forall_cases(INITIAL_POOL_STATES)
    @with_setup(initialize_pool, destroy_pool)
    def setup_case_and_test_it(case):
        load_pool_with_members(case)
        testfn(case)
    setup_case_and_test_it.__name__ = \
        "test_%s_for_a_subscriber_case" % testfn.__name__
    return setup_case_and_test_it

Here’s what the decorator does. When you apply it to a test function testfn, it returns a test generator that yields a property-check test for each of the initial pool states. For each, it sets up a new pool, loads it with the initial subscribers (as given by the corresponding test case), runs the given check function testfn, and then cleans up after itself.

With this decorator, our Pythonic property definitions now mirror the human-language definitions from the start of the article:

@forall_initial_pools
def check_subscribe(case):
    subscribe("1")
    assert "1" in pool_members()

@forall_initial_pools
def check_unsubscribe(case):
    unsubscribe("1")
    assert "1" not in pool_members()

And that’s pretty much the solution I ended up using at work. There, as opposed to here, I got to reuse my decorators for many more tests, making them all the more worth their small implementation price.

Introductory Haskell: Solving the Sorting-It-Out Kata

Tom Moertel — Tue, 31 Oct 2006 14:44:00 -0500

Last Tuesday, my friend Casey and I were hanging out at Aldo Coffee. We planned on enjoying some espresso, doing some work, and then heading over to the Pittsburgh Coding Dojo, where we could hang out with other geekly folks. We ended up not having enough time to go to the meeting, but we decided to hack on the challenge problem anyway, using Aldo’s ever-handy free wireless to access the Internet.

The Dojo problem was PragDave’s Kata Eleven – Sorting it Out. (It’s short; read it now.) We decided to use Haskell for our implementation language.

In this post, I’ll walk through our coding session and explain how our solution evolved. To better fit the session into a blog post, I have removed a lot of back-and-forth micro iterations, and I have edited some of the code for clarity.

The first part of the problem

The first part of the problem was “Sorting Balls.” The story: You need to implement a “rack” to hold the balls drawn at random (without replacement) from a bin containing sixty balls, numbered 0 to 59. Regardless of the order in which the balls are added to the rack, you need to present them in sorted order whenever you’re asked for them.

Upon reading this part of the challenge, a couple of thoughts sprung to mind:

Because the range of balls is so small, the problem was begging for a solution based on a counting sort.
Because the balls are uniquely numbered and drawn without replacement, we could even use a bit vector to represent counts.

Nevertheless, we decided to ignore these thoughts and implement a more-general solution that would work for any (orderable) values, not just small ranges of integers.

Sketching the interface

The first step, then, was to sketch out an interface. Our interface mirrored the one from the problem statement but was tweaked for Haskell:

mkRack :: Rack a
add    :: Ord a => a -> Rack a -> Rack a
balls  :: Rack a -> [a]

The function mkRack makes a new rack to hold values (“balls”) of type a. It’s equivalent to Rack.new in Ruby.

The add function adds a ball to a rack. You give it a ball and a rack, and it returns a new rack that is the same as the original rack but also contains the ball. (If you’re accustomed to stateful programming, this may seem weird. Why return a new rack instead of modifying the original rack? Because, in Haskell, you can’t change values: you can only create new values. At first, this constraint may seem limiting, but after you get used to it, you’ll find it empowering.)

Note: the Ord a qualification on the type signature of add says that it will work for any type a whose values can be ordered. The qualification is necessary because values of some types, like IO actions, cannot be compared to see which are less than the others.

The balls function is an “observer”: it lets you observe the balls in a rack by returning them as an ordered list.

And that’s the interface.

With the interface sketched, we gave it meaning by defining its properties.

Giving our interface meaning: defining properties using QuickCheck

QuickCheck is a powerful, easy-to-use testing tool. Instead of checking test cases, it checks properties – statements about what your code ought to do in general.

The great thing about QuickCheck properties is that they are testable documentation. They tell the world what your code is supposed to do, and they do so in a concise, formal language that just happens to be easily readable by humans and automatically testable by computers.

To specify the desired properties of our Rack interface, we first had to import QuickCheck:

import Test.QuickCheck

Then, we defined our first property. It said, simply, that a new rack must be empty when observed:

prop_New =
    balls mkRack =~ []

Our second property said that, when you add a ball x to a rack, the resulting rack must contain the same balls as the original rack plus x:

prop_AddAddsElement rack x =
    balls (add x rack) =~ (x : balls rack)

Both of the properties above rely upon a special, order-insensitive equality test that we defined for lists of Int values:

(=~) :: [Int] -> [Int] -> Bool
xs =~ ys = sort xs == sort ys

Note that under this test, [1,2] “equals” both [1,2] and [2,1], but it does not “equal” any other values.

The reason we defined this operator was to help us specify the two essential properties of add separately: (1) it must insert a ball into a rack, and (2) the new ball’s position, when observed, must preserve the rack’s ordering invariant. The previous property definition used the =~ operator to specify the first of these two properties. The next property we defined specified the second:

prop_AddPreservesOrdering rack x =
    isOrdered (balls rack) ==> isOrdered (balls (add x rack))

This definition specifies that, for all racks rack and all balls x, if the balls in rack are ordered, the balls in the rack that results from adding x to rack must also be ordered. If you are familiar with proof by induction, you’ll know why we went this route. In short, if we can prove that this property holds (and, trivially, that an empty rack is ordered), we can prove that add preserves the ordering invariant.

To round out the property definition, we needed to define the isOrdered test:

isOrdered :: [Int] -> Bool
isOrdered xs = xs == sort xs

And those are the properties we needed to check the correctness of our implementation. Of course, we still needed to write our implementation, and we turned to that task next.

A simple, list-based Rack implementation

For our first implementation, we decided upon a drop-dead-simple list-based representation. We would keep the elements of the list in sorted order by inserting them into the correct positions when add was called.

Here, then, was our code:

-- Our list-based implementation of a Rack

type Rack a = [a]

mkRack   = []
add x xs = insertList x xs
balls    = id

insertList :: Ord a => a -> [a] -> [a]
insertList x []     = [x]
insertList x (y:ys)
    | x < y         = x : y : ys
    | otherwise     = y : insertList x ys

That’s it.

We took our new implementation for a spin in GHCi:

*Rack> balls mkRack
[]

*Rack> balls (add 3 mkRack)
[3]

*Rack> balls (add 4 (add 3 mkRack))
[3,4]

*Rack> balls (add 1 (add 4 (add 3 mkRack)))
[1,3,4]

*Rack> balls (foldr add mkRack [4,2,6,3,-9,0,33,9])
[-9,0,2,3,4,6,9,33]

To really test our implementation, we asked QuickCheck to check its properties:

*Rack> quickCheck prop_New
OK, passed 100 tests.

*Rack> quickCheck prop_AddAddsElement
OK, passed 100 tests.

*Rack> quickCheck prop_AddPreservesOrdering
OK, passed 100 tests.

I should point out that QuickCheck did not prove that our properties held. Rather, it gathered evidence that we could use to argue that our properties held. The evidence was that each of our properties’ claims was subjected to 100 randomly generated tests, and none of the tests was able to disprove a claim.

Was this evidence sufficient for us to rest satisfied that our implementation was correct? Given how simple our implementation was, I felt that the evidence was sufficient. Casey agreed, and we moved on.

With the first implementation done, we decided to try a more-sophisticated implementation.

Generalizing the interface

Since we were about to have multiple implementations, it made sense for us to define a generalized interface that any “Rack-like” implementation could use. For that, Haskell’s type classes were perfect:

-- Our interface for "Rack-like" data types

class Racklike a ra | ra -> a where
    mkRack :: ra
    add    :: Ord a => a -> ra -> ra
    balls  :: ra -> [a]

The interface was essentially the same as before, except that the data type behind the rack implementation was not given by a specific type Rack a but rather by the type variable ra, which represents some type of rack container for balls of type a.

Note that ra determines a. If, for example, you know that the container type ra equals “a list of Int values,” you know that a must equal Int. (To represent this relationship, we used functional dependencies, a popular extension to the Haskell 98 standard.)

With the Racklike type class in place, we moved our list-based implementation inside of the interface:

-- Our list-based implementation of a Rack

type ListRack a = [a]

instance Racklike a (ListRack a) where
    mkRack = []
    add    = insertList
    balls  = id

Next, we modified our QuickCheck property definitions. Where before it was fine to assume that we would be testing our single, list-based implementation, now we needed to allow for testing other implementation types. We did this by adding a rackType parameter to our property definitions. We used the type, not the value, of this parameter to determine the type of rack to test:

prop_New rackType =
    balls (mkRack `asTypeOf` rackType) =~ []

prop_AddAddsElement rackType ballList x =
    balls (add x rack) =~ (x : balls rack)
  where
    rack = rackFromList ballList `asTypeOf` rackType

prop_AddPreservesOrdering rackType ballList x =
    isOrdered (balls rack) ==> isOrdered (balls (add x rack))
  where
    rack = rackFromList ballList `asTypeOf` rackType

Because we could no longer assume the rack would be represented as a list of integers, we wrote rackFromList to convert such a list into a rack:

rackFromList xs = foldr add mkRack xs

With these modifications in place, we re-ran our tests, specifying (via type annotations) that we wanted to run them for the ListRack implementation:

*Rack> quickCheck $ prop_New (undefined :: ListRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddAddsElement (undefined :: ListRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddPreservesOrdering (undefined :: ListRack Int)
OK, passed 100 tests.

A tree-based Rack implementation

Now that we were free to add additional implementation types, we created one based on binary trees. We started by defining the tree data type:

data Tree a
    = Empty
    | Root (Tree a) a (Tree a)
    deriving (Ord, Eq, Show)

This definition says that a tree can be either empty or a root node. A root node has a single value and left and right sub-trees.

Further, root nodes must satisfy an ordering invariant: if a root node’s value is x, all of the values in its left subtree must be less than x, and all of the values in its right subtree must be greater than or equal to x. The data type doesn’t enforce this invariant, so we would need to enforce it in our implementation.

Next, we wrote the basic functions for creating, adding elements to, and observing our trees.

We needed to be able to create empty trees:

emptyTree =
    Empty

Inserting an element into a tree requires us to walk the tree and append the element as a new leaf node in the correct location, being mindful of our ordering invariant. Because our data structure is inherently recursive, a recursive implementation was straightforward to code:

insertTree x Empty  = Root Empty x Empty
insertTree x (Root left y right)
    | x < y         = Root (insertTree x left) y right
    | otherwise     = Root left y (insertTree x right)

Note that we don’t try to ensure that the tree is balanced. The problem statement says that the balls are randomly selected, and thus we can expect our trees, on average, to be balanced naturally.

Next, we wrote the code to observe the elements of a tree. We used a functional-programming idiom for efficiently flattening a tree into a list:

elemsTree rx =
    elemsTree' rx []

elemsTree' Empty               = id
elemsTree' (Root left x right) =
    elemsTree' left . (x :) . elemsTree' right

Finally, we defined a new tree-based rack type and declared it to be an instance of the Racklike type class:

type TreeRack a = Tree a

instance Racklike a (TreeRack a) where
    mkRack = emptyTree
    add    = insertTree
    balls  = elemsTree

With the implementation done, we took it for a test drive:

*Rack> add 1 mkRack :: TreeRack Int
Root Empty 1 Empty

*Rack> add 3 (add 1 mkRack) :: TreeRack Int
Root Empty 1 (Root Empty 3 Empty)

*Rack> balls (add 3 (add 1 mkRack) :: TreeRack Int)
[1,3]

Then, for the real test, we checked that our properties held for TreeRacks:

*Rack> quickCheck $ prop_New (undefined :: TreeRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddAddsElement (undefined :: TreeRack Int)
OK, passed 100 tests.

quickCheck $ prop_AddPreservesOrdering (undefined :: TreeRack Int)
OK, passed 100 tests.

Satisfied with these results, we moved on to part two of the problem.

The second part of the problem

The second part of the problem was about sorting the letters within a block of text, ignoring white space and punctuation, and converting upper case letters into lower case: “Are there any ways to perform this sort cheaply, and without using built-in libraries?”

Again, a counting sort seemed like an obvious ideal solution, but we decided to recycle our existing code since we had to leave soon. Because our Rack implementations were generic, they would work on letters just as well as on numbers or other kinds of balls:

*Rack> balls (rackFromList "this is a test" :: TreeRack Char)
"   aehiisssttt"

With our existing code already doing the hard work for us, it was trivial to code up the letter-sorting function:

sortLetters xs =
    balls (rackFromList letters :: TreeRack Char)
  where
    letters = [toLower x | x <- xs, isAlpha x]

(Note: Because of the nature of the problem, I interpreted the question’s “without using built-in libraries” to mean “without built-in sorting libraries.”)

We took the new function for a test drive, and it worked as expected:

*Rack> sortLetters "This is a test, pal." 
"aaehiilpsssttt"

And that ended our coding session.

Update: Tweaked the revised definition of the AddAddsElement property for greater parallelism with the original.

Update 2007-03-03: Minor edits for clarity.