Tom Moertel's Weblog: Tag python

Thinking algebraically: a functional-programming dividend that pays during your imperative-programming day job

Tom Moertel — Wed, 20 Aug 2008 21:50:00 -0400

Although at work I code mostly in Python – a language from which lambda and map were nearly removed – I still find that functional-programming experience has its benefits. One of the “functional-programming dividends” I notice most often is insight gained from considering problems from an algebraic perspective.

Recently, for example, I had a small parsing problem. I had to write code that, given a simple grammar specification as input, emits a regular expression that matches the language generated by the grammar.

Here’s a simplified version of the problem. A grammar specification is limited to a series of one or more atoms. For example, “a b c” represents the atom “a”, followed by the atom “b”, followed by the atom “c”. To generate the grammar, the series of atoms is interpreted such that each atom (except the last) generates a production rule of the following form:

atom_rule ::=
  <the literal atom> (SPACE <the next rule> | NOTHING)

(SPACE represents literal white space and NOTHING represents an empty string.) The grammar as a whole is rooted in the first atom’s rule.

Thus the specification “a b c” represents the following grammar:

grammar ::= a_rule
a_rule  ::= "a" (SPACE b_rule | NOTHING)
b_rule  ::= "b" (SPACE c_rule | NOTHING)
c_rule  ::= "c"

Note that the final atom’s production matches only the literal atom itself: it has no following rule on which to chain.

The grammar, in turn, generates the following language:

a
a b
a b c

Thus, given the grammar specification “a b c”, my code had to produce a regular expression that would match “a”, “a b”, or “a b c”.

At this point, please stop for a moment and think about this little programming exercise. Do any solutions leap to mind? How would you approach the problem? Form your opinions now, because I’m going to ask you about them later. (If you’re feeling especially caffeinated, try coding a solution before reading on.)

For me, the insight that made the exercise easy was seeing that the grammar is given by folding a (suitably defined) right-associative binary operator through the series of atoms. The relationship might be easier to see if you substitute away the intermediate production rules from the grammar above:

grammar ::= "a" (SPACE "b" (SPACE "c" | NOTHING) | NOTHING)

If you squint past the SPACE and NOTHING terms, you’ll see that the grammar has the form

(a + (b + (c)))

The + is a binary operator that generates the parts we squinted away. Once you see what’s going on structurally, the operator is easy to define:

x + y = x (SPACE y | NOTHING)

Compare the operator’s definition with that of the atom_rule I presented at the beginning of the article. They’re structurally the same: the operator’s x and y are the atom rule’s <the literal atom> and <the next rule>.

Now all that remains is to generalize the “a b c” formula into a general formula that works for arbitrary grammar specifications. Fortunately, this work has already been done for us. The generalized formula is nothing more than a right fold. In Haskell, the particular right-fold flavor we want is called foldr1.

Given a list of atoms, we can use foldr1 to construct its grammar as follows:

mk_grammar atoms = foldr1 (+) atoms

But Python, our implementation language, does not offer a foldr1 function. This wrinkle, however, is another thing we can iron out by thinking algebraically. Python doesn’t have foldr1, but it does have a reduce function, which represents a left fold, equivalent to Haskell’s foldl’ or foldl1’. Because our + operator is strict and our list of atoms is finite, we can take advantage of the following identity:

foldr1 (+) xs == foldl1 (flip (+)) (reverse xs)

That is, we can convert a right fold into a left fold by flipping the arguments of the operator and operating on the list in reverse. Thus we can implement the fold we want in terms of the fold we have:

# Python code

def foldr1(f, xs):
    return reduce(flip(f), reversed(xs))

def flip(f):
    def g(x, y):
        return f(y, x)
    return g

Now writing a Python-based solution is straightforward:

def grammar_spec_to_re(spec):
    atoms = grammar_spec_to_atoms(spec)
    atom_res = map(atom_to_re, atoms)
    grammar_re = r'\A%s\Z' % foldr1(op, atom_res)
    return grammar_re

def op(x, y):
    # x + y = x (SPACE y | NOTHING)
    return r'%s(\s+%s)?' % (x, y)

def grammar_spec_to_atoms(spec):
    return spec.split()

def atom_to_re(atom):
    return re.escape(atom)

Using our solution, let’s compile the “a b c” grammar specification into its corresponding regular expression:

>>> print grammar_spec_to_re('a b c')
\Aa(\s+b(\s+c)?)?\Z

And that’s basically how I solved the problem.

To play around with the solution, here’s a small helper class that compiles a grammar specification into a regular expression and then tests strings for matching the regexp:

class GrammarMatcher(object):
    def __init__(self, spec):
        self.re = re.compile(grammar_spec_to_re(spec))
    def __call__(self, s):
        return not not self.re.match(s)

Now, let’s try out the regular expression generated for the grammar specification “a b c” :

>>> matcher = GrammarMatcher('a b c')
>>> matcher('')
False
>>> matcher('a')
True
>>> matcher('ab')
False
>>> matcher('a b')
True
>>> matcher('a  b')
True
>>> matcher('a b c')
True
>>> matcher('a b c d')
False
>>> matcher('a c')
False
>>> matcher('b')
False

Now, those questions I promised. If you’re a functional programmer, did a fold-based solution leap out at you? (Did you think of the problem in terms of folds?) If you’re not a functional programmer, how did you see the problem? Did the solution above seem twisted, confusing, or overly clever?

(There are no right or wrong answers. I’m just curious about how people with different backgrounds view the problem.)

Update: Edited to clarify that the problem is to convert a grammar specification into a regular expression, not just test whether a string matches a specified grammar.

Property checking with Python's nose testing framework

Tom Moertel — Wed, 19 Mar 2008 22:34:00 -0400

At work recently I was writing some tests with Python’s out-of-the-box unit-testing framework unittest. I’m new to Python and accustomed to Perl and Haskell’s testing frameworks, which are lightweight and let you write tests without much hoop-jumping. In particular, QuickCheck and LectroTest make it easy to test at the property level instead of the test-case level. With unittest, I was having to write a lot of code to get the same level of abstraction.

By “property level,” here’s what I mean. Say I’m testing this thing, let’s call it a subscriber pool. It has two fundamental properties:

Subscribe. For all initial states of the pool, if you call subscribe(user), then, assuming there have been no other operations on the pool, user must be in the pool.
Unsubscribe. For all initial states of the pool, if you call unsubscribe(user), then, assuming there have been no other operations on the pool, user must not be in the pool.

That’s it. If my implementation satisfies both properties, it’s correct. (This is a simplified version of my real testing problem, which required additional property checks.)

To test whether my implementation satisfies each property, I must write individual test cases that together “cover” the property. For example, to test whether the Subscribe property holds, I might write four test cases:

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testEmpty(self):
        load_pool_with_members([])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testOtherGuyAlreadyInPool(self):
        load_pool_with_members(["2"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAlreadyInPool(self):
        load_pool_with_members(["1"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAndOtherGuyAlreadyInPool(self):
        load_pool_with_members(["1", "2"])
        subscribe("1")
        self.assert_("1" in pool_members())

Every one of the test cases has the same form. The repetition makes me want to refactor the whole thing.

Okay, let’s do it:

INITIAL_POOL_STATES = [[], ["2"], ["1"], ["1", "2"]]

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

We’re fighting a bit with the testing framework because our notion of when set-up and tear-down should occur doesn’t match its own, but otherwise our code is looking much more manageable. In particular, if we want to extend our property-check coverage with additional initial pool states, we don’t need to write additional tests; instead, we can just extend a single list.

But we’re only halfway done. We must also check the Unsubscribe property. The code for it is virtually the same as for Subscribe, but with subscribe becoming unsubscribe and in becoming not in. Let’s add it to our class:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

    def testUnsubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                unsubscribe("1")
                self.assert_("1" not in pool_members(), case)
            finally:
                self.tearDown()

And now let’s factor out the new redundancy:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        def testfn(case):
            subscribe("1")
            self.assert_("1" in pool_members(), case)
        self._forall_test_cases(testfn)

    def testUnsubscribe(self):
        def testfn(case):
            unsubscribe("1")
            self.assert_("1" not in pool_members(), case)
        self._forall_test_cases(testfn)

    def _forall_test_cases(self, testfn):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                testfn(case)
            finally:
                self.tearDown()

It’s not bad, but it’s not great either. There’s still a lot of noise in that code.

After discussing the situation with my more-Pythonic colleague Tim Lesher, I took his advice to check out the nose testing framework.

One of the things I liked right away about nose was that it supports test generators, which would let me represent each property-check as a generator that yields the test cases needed to check the property. Also, set-up and tear-down would automatically occur per generated test, so I wouldn’t have to invoke them manually.

Once I got familiar with nose, it was easy to create a decorator to represent the forall-test-cases idiom:

def forall_cases(cases):
    def decorate(testfn):
        def gen():
            for case in cases:
                yield testfn, case
        gen.__name__ = "test_%s_for_a_case" % testfn.__name__
        return gen
    return decorate

Note that this decorator is not specific to our subscriber-pool tests. It can be used in any situation where we need to check a property across a collection of cases. In fact, I keep this little gem in a “nosehelpers” library, where I reuse it all the time. Here’s an example of how to use it to check the trivial property that x = x for all x in 0–99:

@forall_cases(range(100))
def check_self_equality(x):
    assert x == x

Now, back to our testing problem. Here’s how we can use the decorator to check the Subscribe property:

@forall_cases(INITIAL_POOL_STATES)
@with_setup(initialize_pool, destroy_pool)
def check_subscribe(case):
    load_pool_with_members(case)
    subscribe("1")
    assert "1" in pool_members()

(The with_setup decorator is defined by nose and tells nose to run the given set-up and tear-down functions before and after each of the generated test cases.)

Not bad. The only problem I have with that code is that it mixes the “For all initial states of the pool” part of the property definition into the “if you call subscribe(user), then …” part. I’d like the code to be more explicit about which part defines the scope of the property claim and which part defines the test for whether the claim holds for any particular test case within that scope.

Fortunately, we can build upon our existing decorator to create exactly what we need:

def forall_initial_pools(testfn):
    @forall_cases(INITIAL_POOL_STATES)
    @with_setup(initialize_pool, destroy_pool)
    def setup_case_and_test_it(case):
        load_pool_with_members(case)
        testfn(case)
    setup_case_and_test_it.__name__ = \
        "test_%s_for_a_subscriber_case" % testfn.__name__
    return setup_case_and_test_it

Here’s what the decorator does. When you apply it to a test function testfn, it returns a test generator that yields a property-check test for each of the initial pool states. For each, it sets up a new pool, loads it with the initial subscribers (as given by the corresponding test case), runs the given check function testfn, and then cleans up after itself.

With this decorator, our Pythonic property definitions now mirror the human-language definitions from the start of the article:

@forall_initial_pools
def check_subscribe(case):
    subscribe("1")
    assert "1" in pool_members()

@forall_initial_pools
def check_unsubscribe(case):
    unsubscribe("1")
    assert "1" not in pool_members()

And that’s pretty much the solution I ended up using at work. There, as opposed to here, I got to reuse my decorators for many more tests, making them all the more worth their small implementation price.

How to download photos and movies from the Palm Centro to a Linux desktop

Tom Moertel — Fri, 02 Nov 2007 15:32:00 -0400

I recently got a Palm Centro smartphone, and so far I love it. Like most modern cell phones, it has a built-in camera and takes decent snapshots and even records short movies. It’s great for spur-of-the-moment shots when I don’t have my real camera. The trick – and there’s always a trick when it comes to cell phones – is getting the photos off the camera and onto my computer.

To get at my pictures, Sprint would prefer that I sign up for their ludicrously expensive “PictureMail” service. Leave it to weasely telecom execs to come up with another way to squeeze money from teenagers: charge them $5 each month for the “privilege” of sharing their pictures with friends. This fee, of course, is in addition to the fee for “unlimited” mobile Internet use. I guess picture bits are somehow more expensive to move over the air than other kinds of bits.

In any case, my next goal after getting my Centro to hotsync with my Linux workstation was to figure out how to download my photos and movies.

After a bit of hacking, I figured out that the Centro stores images in a typical digital-camera-image (DCIM) hierarchy. For example, I have a 4-GB microSD card installed in my Centro, and I store my photos in the “Palm” album on it. This album ends up stored in the /DCIM/Palm directory on the card.

Using the pilot-xfer program from the pilot-link project, I was able to find the directory and its contents. The trick was to use the sparsely documented –D flag to work with the Centro’s virtual filesystem. Here, for example, is how I list the contents of the Palm album:

$ pilot-xfer -p usb: -D /DCIM/Palm -l

   Listening for incoming connection on usb:... connected!

   Directory of /DCIM/Palm...
        652 Fri Nov  2 08:17:06 2007  Album.db
     292053 Fri Nov  2 09:04:20 2007  Photo_110207_001.jpg
      78493 Fri Nov  2 08:17:06 2007  Video_110207_001.3g2
         20 Wed Oct 31 12:09:20 2007  Thumbnail.db

   Thank you for using pilot-link.

Here, you can see that I have one photo and one movie in the album. (Movies are stored in .3g2 files that contain MPEG4 video.)

To download the files, I again turned to pilot-xfer, this time using the –f (fetch) flag to fetch a list of files. Here, for example, I’ll fetch the image from the listing above:

$ pilot-xfer -p usb: -D /DCIM/Palm -f Photo_110207_001.jpg

   Listening for incoming connection on usb:... connected!

   Fetching '/DCIM/Palm' ... (292053 bytes)   285 KiB total.

   Thank you for using pilot-link.

So that’s the process. It’s kind of clunky, so I wrote a small Python program to automate it. (I’m learning Python. If you’re a Pythonista, please consider critiquing my code. I would be especially thankful if you could point out any helpful idioms that I may have overlooked.)

Here’s how to use the program:

$ get-pilot-photos.py --help
Usage: get-pilot-photos.py [options]

Options:
  -h, --help            show this help message and exit
  -s SRCDIR, --srcdir=SRCDIR
                        VFS dir on Palm device from which to fetch images
  -d DESTDIR, --destdir=DESTDIR
                        Where to save the images on your computer

Both the —srcdir and —dstdir options are optional. If you omit the first, the program will download photos and movies from the /DCIM/Palm album. If you omit the second, the program will save the downloads to a new, timestamped directory within your home directory.

That’s it. The code is below.

#!/usr/bin/env python

# get-pilot-photos.py -
# Download photos and movies from my Palm Centro via pilot-link
#
# Tom Moertel <tom@moertel.com>
# 2007-11-01
#
# Copyright 2007 Thomas G. Moertel
#
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# See <http://www.gnu.org/licenses/> for more.

import os
import optparse
import re
import subprocess
import time

PILOT_XFER = 'pilot-xfer'
DEFAULT_PALM_IMAGE_DIR = '/DCIM/Palm'

class PhotoImporter(object):

    def __init__(self, src_dir, dest_dir=None):
        self.src_dir = src_dir
        self.dest_dir = dest_dir or self.get_image_dir()

    def run(self):
        print 'Finding images in %s on your Palm device.' % self.src_dir
        print 'Begin hotsync now...'
        images = self.get_image_list()
        if len(images) == 0:
            print 'No images were found.  Done.'
            return
        print 'Found %s images' % len(images)
        print 'Waiting for hotsync to complete...'
        time.sleep(10) # give 1st hotsync time to complete
        print 'Begin another hotsync now...'
        self.fetch_images(images)
        print 'Done.  The images were fetched to the following directory:'
        print self.dest_dir

    def get_image_list(self):
        cmdline = [PILOT_XFER, '-p',  'usb:', '-D', self.src_dir, '-l']
        proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE)
        listing = proc.stdout.read()
        proc.wait()
        return re.findall(r'\b\S+\.(?:jpg|3g2)\b', listing)

    def fetch_images(self, images):
        cmdline = [PILOT_XFER, '-p',  'usb:', '-D', self.src_dir, '-f'] + images
        subprocess.Popen(cmdline, cwd=self.dest_dir).wait()

    def get_image_dir(self):
        root = os.getenv('HOME') or tempfile.mkdtemp()
        now = time.strftime("%Y-%m-%d--%H.%M.%S", time.localtime())
        dir = os.path.join(root, 'images', 'unsorted-pix', now)
        os.makedirs(dir, mode=0771)
        return dir

def main():
    p = optparse.OptionParser()
    p.add_option('--srcdir', '-s', default=DEFAULT_PALM_IMAGE_DIR,
                 help='VFS dir on Palm device from which to fetch images')
    p.add_option('--destdir', '-d',
                 help='Where to save the images on your computer')
    opts, args = p.parse_args()
    PhotoImporter(opts.srcdir, opts.destdir).run()

if __name__ == '__main__':
    main()