Tom Moertel's Weblog

Property checking with Python's nose testing framework

Posted by Tom Moertel Thu, 20 Mar 2008 02:34:00 GMT

At work recently I was writing some tests with Python’s out-of-the-box unit-testing framework unittest. I’m new to Python and accustomed to Perl and Haskell’s testing frameworks, which are lightweight and let you write tests without much hoop-jumping. In particular, QuickCheck and LectroTest make it easy to test at the property level instead of the test-case level. With unittest, I was having to write a lot of code to get the same level of abstraction.

By “property level,” here’s what I mean. Say I’m testing this thing, let’s call it a subscriber pool. It has two fundamental properties:

1. Subscribe. For all initial states of the pool, if you call subscribe(user), then, assuming there have been no other operations on the pool, user must be in the pool.
2. Unsubscribe. For all initial states of the pool, if you call unsubscribe(user), then, assuming there have been no other operations on the pool, user must not be in the pool.

That’s it. If my implementation satisfies both properties, it’s correct. (This is a simplified version of my real testing problem, which required additional property checks.)

To test whether my implementation satisfies each property, I must write individual test cases that together “cover” the property. For example, to test whether the Subscribe property holds, I might write four test cases:

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testEmpty(self):
        load_pool_with_members([])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testOtherGuyAlreadyInPool(self):
        load_pool_with_members(["2"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAlreadyInPool(self):
        load_pool_with_members(["1"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAndOtherGuyAlreadyInPool(self):
        load_pool_with_members(["1", "2"])
        subscribe("1")
        self.assert_("1" in pool_members())

Every one of the test cases has the same form. The repetition makes me want to refactor the whole thing.

Okay, let’s do it:

Posted in testing
Tags nose, properties, python, testing, unittest
1 comment
no trackbacks
 

Introductory Haskell: Solving the Sorting-It-Out Kata

Posted by Tom Moertel Tue, 31 Oct 2006 19:44:00 GMT

Last Tuesday, my friend Casey and I were hanging out at Aldo Coffee. We planned on enjoying some espresso, doing some work, and then heading over to the Pittsburgh Coding Dojo, where we could hang out with other geekly folks. We ended up not having enough time to go to the meeting, but we decided to hack on the challenge problem anyway, using Aldo’s ever-handy free wireless to access the Internet.

The Dojo problem was PragDave’s Kata Eleven – Sorting it Out. (It’s short; read it now.) We decided to use Haskell for our implementation language.

In this post, I’ll walk through our coding session and explain how our solution evolved. To better fit the session into a blog post, I have removed a lot of back-and-forth micro iterations, and I have edited some of the code for clarity.

The first part of the problem

The first part of the problem was “Sorting Balls.” The story: You need to implement a “rack” to hold the balls drawn at random (without replacement) from a bin containing sixty balls, numbered 0 to 59. Regardless of the order in which the balls are added to the rack, you need to present them in sorted order whenever you’re asked for them.

Upon reading this part of the challenge, a couple of thoughts sprung to mind:

• Because the range of balls is so small, the problem was begging for a solution based on a counting sort.
• Because the balls are uniquely numbered and drawn without replacement, we could even use a bit vector to represent counts.

Nevertheless, we decided to ignore these thoughts and implement a more-general solution that would work for any (orderable) values, not just small ranges of integers.

Sketching the interface

The first step, then, was to sketch out an interface. Our interface mirrored the one from the problem statement but was tweaked for Haskell:

mkRack :: Rack a
add    :: Ord a => a -> Rack a -> Rack a
balls  :: Rack a -> [a]

The function mkRack makes a new rack to hold values (“balls”) of type a. It’s equivalent to Rack.new in Ruby.

The add function adds a ball to a rack. You give it a ball and a rack, and it returns a new rack that is the same as the original rack but also contains the ball. (If you’re accustomed to stateful programming, this may seem weird. Why return a new rack instead of modifying the original rack? Because, in Haskell, you can’t change values: you can only create new values. At first, this constraint may seem limiting, but after you get used to it, you’ll find it empowering.)

Note: the Ord a qualification on the type signature of add says that it will work for any type a whose values can be ordered. The qualification is necessary because values of some types, like IO actions, cannot be compared to see which are less than the others.

The balls function is an “observer”: it lets you observe the balls in a rack by returning them as an ordered list.

And that’s the interface.

With the interface sketched, we gave it meaning by defining its properties.

Giving our interface meaning: defining properties using QuickCheck

QuickCheck is a powerful, easy-to-use testing tool. Instead of checking test cases, it checks properties – statements about what your code ought to do in general.

The great thing about QuickCheck properties is that they are testable documentation. They tell the world what your code is supposed to do, and they do so in a concise, formal language that just happens to be easily readable by humans and automatically testable by computers.

To specify the desired properties of our Rack interface, we first had to import QuickCheck:

import Test.QuickCheck

Then, we defined our first property. It said, simply, that a new rack must be empty when observed:

prop_New =
    balls mkRack =~ []

Our second property said that, when you add a ball x to a rack, the resulting rack must contain the same balls as the original rack plus x:

prop_AddAddsElement rack x =
    balls (add x rack) =~ (x : balls rack)

Both of the properties above rely upon a special, order-insensitive equality test that we defined for lists of Int values:

(=~) :: [Int] -> [Int] -> Bool
xs =~ ys = sort xs == sort ys

Note that under this test, [1,2] “equals” both [1,2] and [2,1], but it does not “equal” any other values.

The reason we defined this operator was to help us specify the two essential properties of add separately: (1) it must insert a ball into a rack, and (2) the new ball’s position, when observed, must preserve the rack’s ordering invariant. The previous property definition used the =~ operator to specify the first of these two properties. The next property we defined specified the second:

prop_AddPreservesOrdering rack x =
    isOrdered (balls rack) ==> isOrdered (balls (add x rack))

This definition specifies that, for all racks rack and all balls x, if the balls in rack are ordered, the balls in the rack that results from adding x to rack must also be ordered. If you are familiar with proof by induction, you’ll know why we went this route. In short, if we can prove that this property holds (and, trivially, that an empty rack is ordered), we can prove that add preserves the ordering invariant.

To round out the property definition, we needed to define the isOrdered test:

isOrdered :: [Int] -> Bool
isOrdered xs = xs == sort xs

And those are the properties we needed to check the correctness of our implementation. Of course, we still needed to write our implementation, and we turned to that task next.

A simple, list-based Rack implementation

For our first implementation, we decided upon a drop-dead-simple list-based representation. We would keep the elements of the list in sorted order by inserting them into the correct positions when add was called.

Here, then, was our code:

-- Our list-based implementation of a Rack

type Rack a = [a]

mkRack   = []
add x xs = insertList x xs
balls    = id

insertList :: Ord a => a -> [a] -> [a]
insertList x []     = [x]
insertList x (y:ys)
    | x < y         = x : y : ys
    | otherwise     = y : insertList x ys

That’s it.

We took our new implementation for a spin in GHCi:

*Rack> balls mkRack
[]

*Rack> balls (add 3 mkRack)
[3]

*Rack> balls (add 4 (add 3 mkRack))
[3,4]

*Rack> balls (add 1 (add 4 (add 3 mkRack)))
[1,3,4]

*Rack> balls (foldr add mkRack [4,2,6,3,-9,0,33,9])
[-9,0,2,3,4,6,9,33]

To really test our implementation, we asked QuickCheck to check its properties:

*Rack> quickCheck prop_New
OK, passed 100 tests.

*Rack> quickCheck prop_AddAddsElement
OK, passed 100 tests.

*Rack> quickCheck prop_AddPreservesOrdering
OK, passed 100 tests.

I should point out that QuickCheck did not prove that our properties held. Rather, it gathered evidence that we could use to argue that our properties held. The evidence was that each of our properties’ claims was subjected to 100 randomly generated tests, and none of the tests was able to disprove a claim.

Was this evidence sufficient for us to rest satisfied that our implementation was correct? Given how simple our implementation was, I felt that the evidence was sufficient. Casey agreed, and we moved on.

With the first implementation done, we decided to try a more-sophisticated implementation.

Generalizing the interface

Since we were about to have multiple implementations, it made sense for us to define a generalized interface that any “Rack-like” implementation could use. For that, Haskell’s type classes were perfect:

-- Our interface for "Rack-like" data types

class Racklike a ra | ra -> a where
    mkRack :: ra
    add    :: Ord a => a -> ra -> ra
    balls  :: ra -> [a]

The interface was essentially the same as before, except that the data type behind the rack implementation was not given by a specific type Rack a but rather by the type variable ra, which represents some type of rack container for balls of type a.

Note that ra determines a. If, for example, you know that the container type ra equals “a list of Int values,” you know that a must equal Int. (To represent this relationship, we used functional dependencies, a popular extension to the Haskell 98 standard.)

With the Racklike type class in place, we moved our list-based implementation inside of the interface:

-- Our list-based implementation of a Rack

type ListRack a = [a]

instance Racklike a (ListRack a) where
    mkRack = []
    add    = insertList
    balls  = id

Next, we modified our QuickCheck property definitions. Where before it was fine to assume that we would be testing our single, list-based implementation, now we needed to allow for testing other implementation types. We did this by adding a rackType parameter to our property definitions. We used the type, not the value, of this parameter to determine the type of rack to test:

prop_New rackType =
    balls (mkRack `asTypeOf` rackType) =~ []

prop_AddAddsElement rackType ballList x =
    balls (add x rack) =~ (x : balls rack)
  where
    rack = rackFromList ballList `asTypeOf` rackType

prop_AddPreservesOrdering rackType ballList x =
    isOrdered (balls rack) ==> isOrdered (balls (add x rack))
  where
    rack = rackFromList ballList `asTypeOf` rackType

Because we could no longer assume the rack would be represented as a list of integers, we wrote rackFromList to convert such a list into a rack:

rackFromList xs = foldr add mkRack xs

With these modifications in place, we re-ran our tests, specifying (via type annotations) that we wanted to run them for the ListRack implementation:

*Rack> quickCheck $ prop_New (undefined :: ListRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddAddsElement (undefined :: ListRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddPreservesOrdering (undefined :: ListRack Int)
OK, passed 100 tests.

A tree-based Rack implementation

Now that we were free to add additional implementation types, we created one based on binary trees. We started by defining the tree data type:

data Tree a
    = Empty
    | Root (Tree a) a (Tree a)
    deriving (Ord, Eq, Show)

This definition says that a tree can be either empty or a root node. A root node has a single value and left and right sub-trees.

Further, root nodes must satisfy an ordering invariant: if a root node’s value is x, all of the values in its left subtree must be less than x, and all of the values in its right subtree must be greater than or equal to x. The data type doesn’t enforce this invariant, so we would need to enforce it in our implementation.

Next, we wrote the basic functions for creating, adding elements to, and observing our trees.

We needed to be able to create empty trees:

emptyTree =
    Empty

Inserting an element into a tree requires us to walk the tree and append the element as a new leaf node in the correct location, being mindful of our ordering invariant. Because our data structure is inherently recursive, a recursive implementation was straightforward to code:

insertTree x Empty  = Root Empty x Empty
insertTree x (Root left y right)
    | x < y         = Root (insertTree x left) y right
    | otherwise     = Root left y (insertTree x right)

Note that we don’t try to ensure that the tree is balanced. The problem statement says that the balls are randomly selected, and thus we can expect our trees, on average, to be balanced naturally.

Next, we wrote the code to observe the elements of a tree. We used a functional-programming idiom for efficiently flattening a tree into a list:

elemsTree rx =
    elemsTree' rx []

elemsTree' Empty               = id
elemsTree' (Root left x right) =
    elemsTree' left . (x :) . elemsTree' right

Finally, we defined a new tree-based rack type and declared it to be an instance of the Racklike type class:

type TreeRack a = Tree a

instance Racklike a (TreeRack a) where
    mkRack = emptyTree
    add    = insertTree
    balls  = elemsTree

With the implementation done, we took it for a test drive:

*Rack> add 1 mkRack :: TreeRack Int
Root Empty 1 Empty

*Rack> add 3 (add 1 mkRack) :: TreeRack Int
Root Empty 1 (Root Empty 3 Empty)

*Rack> balls (add 3 (add 1 mkRack) :: TreeRack Int)
[1,3]

Then, for the real test, we checked that our properties held for TreeRacks:

*Rack> quickCheck $ prop_New (undefined :: TreeRack Int)
OK, passed 100 tests.

*Rack> quickCheck $ prop_AddAddsElement (undefined :: TreeRack Int)
OK, passed 100 tests.

quickCheck $ prop_AddPreservesOrdering (undefined :: TreeRack Int)
OK, passed 100 tests.

Satisfied with these results, we moved on to part two of the problem.

The second part of the problem

The second part of the problem was about sorting the letters within a block of text, ignoring white space and punctuation, and converting upper case letters into lower case: “Are there any ways to perform this sort cheaply, and without using built-in libraries?”

Again, a counting sort seemed like an obvious ideal solution, but we decided to recycle our existing code since we had to leave soon. Because our Rack implementations were generic, they would work on letters just as well as on numbers or other kinds of balls:

*Rack> balls (rackFromList "this is a test" :: TreeRack Char)
"   aehiisssttt" 

With our existing code already doing the hard work for us, it was trivial to code up the letter-sorting function:

sortLetters xs =
    balls (rackFromList letters :: TreeRack Char)
  where
    letters = [toLower x | x <- xs, isAlpha x]

(Note: Because of the nature of the problem, I interpreted the question’s “without using built-in libraries” to mean “without built-in sorting libraries.”)

We took the new function for a test drive, and it worked as expected:

*Rack> sortLetters "This is a test, pal." 
"aaehiilpsssttt" 

And that ended our coding session.

Posted in programming, functional programming, haskell, testing
Tags haskell, kata, quickcheck, sorting, testing
7 comments
no trackbacks
 

A type-based solution to the "strings problem": a fitting end to XSS and SQL-injection holes?

Posted by Tom Moertel Thu, 19 Oct 2006 01:40:00 GMT

Even skilled programmers have a hard time keeping their web applications free of XSS and SQL-injection vulnerabilities. And it shows: a sobering portion of web sites are open to some scary security threats.

Why are so many sites vulnerable to these well-known holes? Probably because it’s insanely hard for programmers to solve the fundamental “strings problem” at the heart of these vulnerabilities. The problem itself is easy to understand, but we humans aren’t equipped to carry out the solution. Simply put, we just plain suck at keeping a bazillion different strings straight in our heads, let alone consistently and reliably rendering their interactions safe whenever they cross paths in a modern web application. It’s easy to say, “just escape the little buggers,” but it’s hard to get it right, every single time.

Computers, on the other hand, are pretty good at keeping track of details by the bucket-full. Wouldn’t it be nice, then, if our programming languages gave us the power to delegate this nasty “strings problem” to our computers, which could then devote their unwavering mechanical precision to grinding the problem out of existence? Isn’t that the kind of thing modern programming languages are supposed to be good at?

I’d like to think the answer to that question is a big, you betcha.

So let’s grab a modern programming language and solve the strings problem.

Let’s solve the strings problem in Haskell

In this article, we will look at one way (among many) to solve the strings problem: by adding Ruby-style string templates to Haskell. These templates support “interpolation” via the usual, convenient #{var} syntax, but here interpolation is type safe. Haskell’s type system will prevent us from inadvertently mixing incompatible string types, and it will detect mistakes at compile time, before they can become live XSS or SQL-injection holes. Further, our solution will offer us these benefits without making us jump through hoops or pay some onerous syntax penalty.

To be more specific, the system offers the following benefits:

• It provides a string-management kernel that lets you create “safe strings” by certifying a regular string as representing either text or a fragment of a known language.
• It allows you to conveniently define new language types for any string-based language that you can provide an escaping rule for (e.g., XML, URLs, SQL, untrusted user input).
• It provides compile-time syntactic sugar (via Template Haskell) that makes working with safe strings as convenient as working with string interpolation in languages like Ruby and Perl.
• It catches and reports (at compile time) the following commonly made programming errors:
  • failing to escape a plain-old-text string before mixing it into a string that represents a language fragment
  • mixing strings that represent fragments of incompatible languages
  • mixing strings that represent fragments of compatible languages in an ambiguous way (the system will force you to disambiguate)

(This is a long one, so grab an espresso, lean back, and read on in style. Also, if you have a smoking jacket, you might want to get it now.)

Posted in programming, programming languages, haskell, ruby, web development, testing, rails
Tags haskell, ruby, strings, testing, types
37 comments
no trackbacks
 

If unit testing can't keep Rails safe from string-escaping problems, what makes you think it will keep your projects safe?

Posted by Tom Moertel Thu, 12 Oct 2006 20:06:00 GMT

Recently I wrote about unit testing being a tool, not a goal in itself. I argued that unit testing was not a reliable way to fight certain kinds of common coding errors and, therefore, that unit testing ought to be supplemented with other tools.

To support my argument, I gave an example of a common, important coding error that unit testing does a bad job of helping programmers control. That error is failing to manage and escape strings properly: the “strings problem.” It is the mother of XSS and SQL-injection security vulnerabilities, not to mention the cause of legions of broken links and bad HTML on the web.

If you think I’m overstating the problem, or if you think that unit testing is a good way of solving it, let me show you how easy it is for even smart developers to get it wrong.

Consider Ruby on Rails, a great framework for developing web applications. Rails has an extensive suite of unit tests, and the Rails development guidelines require that changes to Rails be accompanied by unit tests that “prove [the] change works.”

Now consider that one of Rails’s most-used and most-scrutinized methods – the venerable link_to helper – contains a fundamental string-escaping error:

require 'rubygems'
require_gem 'rails'
include ActionView::Helpers::UrlHelper

url = "http://example.com?ohms_law?volt=1&amp=3" 
puts link_to("TEST", url)

The code, when executed, prints the following HTML snippet:

<a href="http://example.com?ohms_law?volt=1&amp=3">TEST</a>

The HTML snippet represents a hypertext link. The link should point to the URL given in the code, but because the URL was not properly escaped when it was converted into HTML by the link_to helper, the link is broken:

CORRECT:  http://example.com?ohms_law?volt=1&amp=3
LINK_TO:  http://example.com?ohms_law?volt=1&=3
                                             ^ oops

Here’s what’s going on. Because the URL was not escaped, web browsers misinterpret its “amp” parameter as a character-entity reference, which gets gobbled up when the link’s href attribute is parsed. (To see this for yourself, save the output of the Ruby code into an HTML file, open the file with your favorite web browser, and see where the link points.)

Now, how come the unit tests didn’t catch this problem? It turns out, the tests got it wrong, too, by expecting broken output:

# in url_helper_test.rb

def test_link_tag_with_query
  assert_dom_equal \
    "<a href=\"http://www.example.com?q1=v1&amp;q2=v2\">Hello</a>",
    link_to("Hello", "http://www.example.com?q1=v1&amp;q2=v2")
end

The point isn’t that the Rails developers are dumb. The point is that the Rails developers are smart. If they can’t get the strings problem right, even with all their brains and all their unit testing, what reason does any programmer have to think that unit testing is going to solve this problem reliably?

If, then, you want to solve the strings problem – and you really, seriously ought to want to solve the strings problem – you should consider options beyond unit testing.

Posted in programming, testing, rails
Tags escaping, problem, rails, strings, testing
5 comments
no trackbacks
 

Unit testing is a tool, not a goal

Posted by Tom Moertel Tue, 10 Oct 2006 19:23:00 GMT

Via the ever-entertaining programming.reddit.com, I discovered The (Misguided?) Faith in Unit Tests. I don’t agree with the article’s thesis, but the article did hit upon a grain of unfortunate truth: many unit-testing practitioners have a rather cultish aspect to their devotion. More and more, people push unit testing not because it’s an inexpensive way to establish a lot of confidence in our code’s correctness, but because it’s “good” or “a best practice” or “professional.”

When unit testing becomes an end in itself, something has gone haywire. Software-development practices, including unit testing, are just tools. No one tool is the solution to all problems. To the extent that we over-apply any particular tool, we cheat ourselves. We miss out on the opportunity to use other, more effective tools.

My favorite example of a common, important kind of problem that unit testing isn’t much help for is the “strings problem.” All those cross-site-scripting vulnerabilities and SQL-injection holes you have been reading about in recent days are prime examples. In short, the strings problem is having to keep track of and escape, as needed, the bazillion types of strings that flow as life-blood within the arteries of modern web applications.

Trying to detect string-escaping problems via unit testing is costly because it’s at least as difficult to write the tests correctly as it is to write the code the tests are testing. Further, a programmer who forgets that he needs to escape user-supplied strings before stuffing them into live web pages isn’t going to remember to write tests for his having forgotten to escape the strings.

If, then, you come to the programming party with the preconceived notion that unit testing is the only way to go, there’s a pretty good chance that your web applications are going to be sporting some XSS or SQL-injection holes. On the other hand, if you view unit testing as a tool, you’re likely to consider the possibility that other tools might be more effective for something like the strings problem. You might, for example, decide to use a type-system-based solution, which pretty much makes the strings problem trivial to solve. (I’ll write about solving the strings problem in Haskell in an upcoming article.)

I’m not saying that unit testing is “bad” or that it’s not “good.” What I’m saying is that unit testing is not the goal. The goal is having confidence in your code doing what it ought to do. Unit testing is often an inexpensive way to achieve much of that confidence, but for many situations, the strings problem being one of them, it pays to keep an open mind about supplementing unit testing with other tools.

Posted in programming, testing
Tags culture, testing
no comments
no trackbacks
 

LectroTest: new release, new talk, and the new LectroTest Emporium!

Posted by Tom Moertel Tue, 27 Jun 2006 18:21:00 GMT

I have a bunch of LectroTest news. LectroTest, as you may know, is a specification-based, automatic testing system for Perl. It may look like Haskell’s QuickCheck, but it tastes like sweet, sweet Perl.

LectroTest 0.3500 was released

This version adds automatic tools for recording and playing back failures. Using them, you can automatically build regression-testing suites and incorporate them into your testing plan. All it takes is one new line of code:

use Test::LectroTest
    regressions => "regressions.txt";   # <-- that's it!

See the docs on CPAN for more.

My thanks to Steffen Müller, who suggested the feature and is already using it in cool stuff such as Number::WithError.

Slides from “Testing Tips with LectroTest” are now online

You can get the slides from my talk to the Pittsburgh Perl Mongers on 2006-06-14 here: Talk / Testing Tips with LectroTest. In the talk, I covered some of the newer LectroTest features, such as regression testing and Test::LectroTest::Compat, which lets you mix LectroTest with other Perl testing modules.

The LectroTest Emporium opens!

I have very little artistic ability. Nevertheless, alarming numbers of people seem to love the fiercely metallic mascot I created for LectroTest.

At the last Perl Mongers meeting, for example, people actually told me (somewhat sternly) I should put the adorable LectroTest Robot on t-shirts. I am now delighted to announce that I have taken their advice:

Introducing: The LectroTest Emporium

Some important points:

• Yes, it’s a CafePress store
• I’m not making any money on these things
• I’m using direct printing, not heat-transfer printing, so the Robot won’t crack, feel stiff, or suffer from a yellowish transfer background. (CafePress has a comparison of the methods if you want the full details.)

Some items I have moral reservations about offering:

• LectroTest Robot Teddy Bear - Who would be so reckless as to allow something as fierce and as powerful as the LectroTest Robot to come into direct contact with a defenseless, cuddly teddy bear?
• LectroTest Robot Baby Bib - Actually, this is a great idea: your infant and the Robot exist in a symbiotic relationship. When your baby gets food all over the bib, the Robot will consume it (using a electrochemical process not entirely dissimilar to our human concept of “digestion”). Thus is the baby cleaned and the Robot fueled. It’s win-win.
• LectroTest Robot Dog T-Shirt - I am fairly certain that the immense weight of the Robot would easily crush any smaller animal. This product strikes me as a very bad idea.

The T-shirts, on the other hand, are the robot’s meow. Check out the full collection at The LectroTest Emporium.

Posted in perl, fun stuff, testing, marketing, talks
Tags lectrotest, perl, testing
1 comment
no trackbacks
 

When worlds collide and then dance: Test::LectroTest meets Test::Builder

Posted by Tom Moertel Tue, 08 Feb 2005 17:00:00 GMT

It seems that LectroTest is picking up in popularity because I am starting to get regular requests and feedback. Recently, two related requests came in regarding something that I had put off: Figuring out how to merge specification-based testing, which samples thousands of trials per property check, with case-based testing, which runs one trial per case.

The rub is that case-base testing in Perl is managed by Test::Builder and a related family of modules that are designed to make that kind of testing easy. One convenience they offer is that calling test functions like cmp_ok not only performs a test, in this case a general comparison, but also reports the result of the test to the test harness.

So what happens if somebody wants to perform a cmp_ok test from within a LectroTest property specification? When the property is checked, LectroTest will test whether the property holds by sampling a thousand random trials (by default), each of which will end up calling cmp_ok. Can you see where this is going? Yup, the test harness will end up seeing one thousand separate tests instead of the single test of the property.

The solution to this problem, as to all problems of distinction, is hackery. The Test::* family of modules ends up filtering all calls to test functions such as cmp_ok down to the Test::Builder module’s ok method. This method does two things. First, it reports the result of the test to the test harness without giving us a chance to say otherwise. (Naughty!) Second, it returns the result of the test back to the original caller. As far as property checks are concerned, the first part is bad, and the second is good.

To get rid of the bad part, I redefine the ok method during property checks. I put the implementation into a new module, Test::LectroTest::Compat, that exports a single function holds. This function is used to inject a property check into a plain-old Test::Simple- or Test::More-style test plan. For example, here’s a test plan that uses the new module:

use Test::More tests => 2;
use Test::LectroTest::Compat;

my $prop_nonnegative = Property {
    ##[ x <- Int, y <- Int ]##
    cmp_ok(my_function( $x, $y ), '>=', 0);
}, name => "my_function output is non-negative" ;

holds( $prop_nonnegative ); # assert that the prop holds
cmp_ok( 0, '<', 1, "trivial 0<1 test" ); # a "normal" assertion
# ... and so on ...

What does holds do when called? It redefines Test::Builder’s ok method, runs the property-check trials, restores ok, and finally reports the property check’s result via the newly-restored ok. From there, Test::Builder takes over and does the magic necessary to incorporate the result into the test session’s TAP output.

The code that does all this is as follows:

sub holds {
    my ($diag_store, $results) = check_property(@_);
    my $success = $results->success;
    (my $name   = $results->summary) =~ s/^.*?- /property /;
    $Test->ok($success, $name);
    my $details = $results->details;
    $details =~ s/^.*?\n//;     # remove summary line
    $details =~ s/^\# /    /mg; # replace commenting w/ indent
    $Test->diag(@$diag_store) if @$diag_store;
    $Test->diag($details) if $details;
    return $success;
}

sub check_property {
    no strict 'refs';
    no warnings;
    my $diag_store = [];
    my $property = shift;
    local *Test::Builder::ok = \&disconnected_ok;
    local *Test::Builder::diag = sub { shift; push @$diag_store, @_ };
    return ( $diag_store, 
             Test::LectroTest::TestRunner->new(@_)->run($property) );
}

sub disconnected_ok { $_[1] ? 1 : 0 }

You can see that there is one extra bit of hackery going on. I also redefine Test::Builder’s diag method to capture any diagnostic output that may be emitted during the trials. Typically, this occurs only when a trial fails, and in that case the output is almost certainly worth passing back to the user in context. To ensure that the user sees it in context, I hold on to the captured output until the property check is complete and then roll it into the normal LectroTest diagnostic output. It looks great:

not ok 3 - property 'x is a natural number' falsified in 2 attempts
#     Failed test (t/compat.t at line 32)
#     '0'
#         >
#     '0'
#     Counterexample:
#     $x = 0;

In the first part you see the typical cmp_ok output that the assertion 0 > 0 failed. The second part is the LectroTest counterexample that shows at what part of the test space the assertion failed.

It takes some complicated footwork, but the resulting dance is beautiful. To see two markedly different testing systems – and in some ways markedly different testing philosophies – working in step with one another is gratifying. I suspect that most real-world testing problems can be solved better by a combination of the two approaches than by either alone. Thus I am especially happy about this integration.

After some more refinement, I am going to incorporate Test::LectroTest::Compat into the main distribution. For now, you can get a copy in the LectroTest/FAQs section of the Community Projects site.

Posted in perl, testing
Tags lectrotest, perl, testing
no comments
no trackbacks
 

LectroTest: An automatic, specification-based testing tool for Perl

Posted by Tom Moertel Fri, 10 Sep 2004 16:00:00 GMT

Just a quick note to mention LectroTest, a Perl project I have been working on recently. It’s inspired by QuickCheck for the Haskell programming language. The motivation is that traditional unit testing requires programmers to spell out each and every case to test, which seems like an awful lot of work, much of which probably isn’t necessary. Even so, the payoff of unit testing is real, and we ought to do it. Thus the goal becomes to reduce the pain while keeping the gain.

Rather than working at the level of individual test cases, why not tell the computer about the shape of the “test space” and let the computer generate the cases automatically? That’s what LectroTest does. You declare properties and assert that they hold over a certain test space. LectroTest then generates random cases from within the space and tests your assertion for each. If LectroTest is able to “break” whatever you’re testing, it emits a counterexample that you can feed back into your code to debug the error. (Counterexamples also make for nice regression tests.)

If you write code in Perl and want another option for testing your code, give LectroTest a look.

Posted in programming, perl, testing
Tags lectrotest, perl, testing
no comments
no trackbacks