Property checking with Python's nose testing framework

Tom Moertel — Wed, 19 Mar 2008 22:34:00 -0400

At work recently I was writing some tests with Python’s out-of-the-box unit-testing framework unittest. I’m new to Python and accustomed to Perl and Haskell’s testing frameworks, which are lightweight and let you write tests without much hoop-jumping. In particular, QuickCheck and LectroTest make it easy to test at the property level instead of the test-case level. With unittest, I was having to write a lot of code to get the same level of abstraction.

By “property level,” here’s what I mean. Say I’m testing this thing, let’s call it a subscriber pool. It has two fundamental properties:

Subscribe. For all initial states of the pool, if you call subscribe(user), then, assuming there have been no other operations on the pool, user must be in the pool.
Unsubscribe. For all initial states of the pool, if you call unsubscribe(user), then, assuming there have been no other operations on the pool, user must not be in the pool.

That’s it. If my implementation satisfies both properties, it’s correct. (This is a simplified version of my real testing problem, which required additional property checks.)

To test whether my implementation satisfies each property, I must write individual test cases that together “cover” the property. For example, to test whether the Subscribe property holds, I might write four test cases:

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testEmpty(self):
        load_pool_with_members([])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testOtherGuyAlreadyInPool(self):
        load_pool_with_members(["2"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAlreadyInPool(self):
        load_pool_with_members(["1"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAndOtherGuyAlreadyInPool(self):
        load_pool_with_members(["1", "2"])
        subscribe("1")
        self.assert_("1" in pool_members())

Every one of the test cases has the same form. The repetition makes me want to refactor the whole thing.

Okay, let’s do it:

INITIAL_POOL_STATES = [[], ["2"], ["1"], ["1", "2"]]

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

We’re fighting a bit with the testing framework because our notion of when set-up and tear-down should occur doesn’t match its own, but otherwise our code is looking much more manageable. In particular, if we want to extend our property-check coverage with additional initial pool states, we don’t need to write additional tests; instead, we can just extend a single list.

But we’re only halfway done. We must also check the Unsubscribe property. The code for it is virtually the same as for Subscribe, but with subscribe becoming unsubscribe and in becoming not in. Let’s add it to our class:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

    def testUnsubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                unsubscribe("1")
                self.assert_("1" not in pool_members(), case)
            finally:
                self.tearDown()

And now let’s factor out the new redundancy:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        def testfn(case):
            subscribe("1")
            self.assert_("1" in pool_members(), case)
        self._forall_test_cases(testfn)

    def testUnsubscribe(self):
        def testfn(case):
            unsubscribe("1")
            self.assert_("1" not in pool_members(), case)
        self._forall_test_cases(testfn)

    def _forall_test_cases(self, testfn):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                testfn(case)
            finally:
                self.tearDown()

It’s not bad, but it’s not great either. There’s still a lot of noise in that code.

After discussing the situation with my more-Pythonic colleague Tim Lesher, I took his advice to check out the nose testing framework.

One of the things I liked right away about nose was that it supports test generators, which would let me represent each property-check as a generator that yields the test cases needed to check the property. Also, set-up and tear-down would automatically occur per generated test, so I wouldn’t have to invoke them manually.

Once I got familiar with nose, it was easy to create a decorator to represent the forall-test-cases idiom:

def forall_cases(cases):
    def decorate(testfn):
        def gen():
            for case in cases:
                yield testfn, case
        gen.__name__ = "test_%s_for_a_case" % testfn.__name__
        return gen
    return decorate

Note that this decorator is not specific to our subscriber-pool tests. It can be used in any situation where we need to check a property across a collection of cases. In fact, I keep this little gem in a “nosehelpers” library, where I reuse it all the time. Here’s an example of how to use it to check the trivial property that x = x for all x in 0–99:

@forall_cases(range(100))
def check_self_equality(x):
    assert x == x

Now, back to our testing problem. Here’s how we can use the decorator to check the Subscribe property:

@forall_cases(INITIAL_POOL_STATES)
@with_setup(initialize_pool, destroy_pool)
def check_subscribe(case):
    load_pool_with_members(case)
    subscribe("1")
    assert "1" in pool_members()

(The with_setup decorator is defined by nose and tells nose to run the given set-up and tear-down functions before and after each of the generated test cases.)

Not bad. The only problem I have with that code is that it mixes the “For all initial states of the pool” part of the property definition into the “if you call subscribe(user), then …” part. I’d like the code to be more explicit about which part defines the scope of the property claim and which part defines the test for whether the claim holds for any particular test case within that scope.

Fortunately, we can build upon our existing decorator to create exactly what we need:

def forall_initial_pools(testfn):
    @forall_cases(INITIAL_POOL_STATES)
    @with_setup(initialize_pool, destroy_pool)
    def setup_case_and_test_it(case):
        load_pool_with_members(case)
        testfn(case)
    setup_case_and_test_it.__name__ = \
        "test_%s_for_a_subscriber_case" % testfn.__name__
    return setup_case_and_test_it

Here’s what the decorator does. When you apply it to a test function testfn, it returns a test generator that yields a property-check test for each of the initial pool states. For each, it sets up a new pool, loads it with the initial subscribers (as given by the corresponding test case), runs the given check function testfn, and then cleans up after itself.

With this decorator, our Pythonic property definitions now mirror the human-language definitions from the start of the article:

@forall_initial_pools
def check_subscribe(case):
    subscribe("1")
    assert "1" in pool_members()

@forall_initial_pools
def check_unsubscribe(case):
    unsubscribe("1")
    assert "1" not in pool_members()

And that’s pretty much the solution I ended up using at work. There, as opposed to here, I got to reuse my decorators for many more tests, making them all the more worth their small implementation price.

Tom Moertel's Weblog: Tag nose

Property checking with Python's nose testing framework