Tom Moertel's Weblog

See you at the Pittsburgh Perl Workshop 2008!

Tom Moertel — Wed, 08 Oct 2008 21:04:00 -0400

The 2008 Pittsburgh Perl Workshop is this weekend! I can’t wait. (BTW, there are still seats available. If you can somehow get yourself to Pittsburgh this weekend, by all means, grab a PPW ticket now.)

I’m on the organizing committee, so I get an advance look at the talks, and I’m continually impressed by the quantity and sheer interestingness of the things that the Perl community has to say. When leading members of a community volunteer their time to talk to you about something they’re passionate about, that something is usually fascinating.

This year is no exception. There are tons of talks I want to see. Check out the schedule for Saturday and Sunday, and you’ll see what I mean. (You’ll note that there are even talks on programming GPUs and adorable BUG embedded hardware.)

In addition to technical talks, there are three courses being offered this year. Daniel Klein is once again leading his From Zero To Perl introductory course, which was widely praised at last year’s PPW. Author and Perl trainer Peter Scott is offering Maintaining Code While Staying Sane, which is all about maintaining legacy code, something most programmers must do for a (surprisingly) large chunk of their careers. Finally, the ever-knowledgeable brian d foy is offering his Mastering Perl course for coders interested in learning how to reliably write professional, enterprise-quality Perl programs. (I think there are openings for some of the classes, too. If you’re interested, click one of the links above and try to grab a spot.)

This year we’re expanding on the Hackathons, too. We actually have allocated a dedicated “Hackathon Room” – and we’ve arranged for freshly ground, freshly brewed coffee all day long to fuel the hacking. :-)

All in all, it’s shaping up to be another fun-filled, festive PPW. I hope to see you there!

The Insecurity of Security Questions: Why I met my wife in CWmKryWzuxCSAnMDuIg.

Tom Moertel — Fri, 19 Sep 2008 11:06:00 -0400

Via Dare Obasanjo’s blog, I learned that the much-publicized cracking of Sarah Palin’s Yahoo! email accounts was accomplished by exploiting the weakness of “security questions”. In short, all the attacker needed to do to convince Yahoo’s computers that he was Palin was answer three questions as if he were Palin:

What’s your birthday?
What’s your Zip code?
Where did you meet your spouse?

The attacker says he obtained the answers to these questions in less than an hour. Everything he needed was already public knowledge, and Google and Wikipedia made that knowledge easy to find.

And that’s why when I sign up for web sites that ask me to provide baseline answers for those annoying security questions, I claim that I met my spouse in CWmKryWzuxCSAnMDuIg. What? You’ve never been there? Well, that’s not surprising. It’s not a real place: it’s a password, randomly generated, and remembered for me by password-management software on my computer.

That’s right. Every time I’m asked to establish my “secret” answer to a security question, I generate a random string and use that. Here’s a script I use:

#!/usr/bin/perl

use MIME::Base64;

open my $random, "/dev/urandom" 
    or die "can't open /dev/urandom";
my $bytes;
read $random, $bytes, 16;
close $random;

my $pw = encode_base64($bytes);
$pw =~ tr/A-Za-z0-9//cd;
print "$pw$/";

Then I store the string in my password-management software, just in case the web site asks me for it later. Which should only happen if I forget my primary password for that site. Which should only happen if I can’t get into my password-management software. Which should only happen if I’m totally screwed, anyway, so what are the security questions buying me again?

In sum, if you care about your security, you’re probably picking good passwords already. In that case, security questions can’t help you, but they can harm you by making it easier for an attacker bypass your passwords. That’s how the Palin-email cracker did it. So treat your answers to security questions as if they were passwords – in effect, that’s what they are.

PXSL Tools now on Hackage and GitHub

Tom Moertel — Sun, 24 Aug 2008 22:56:00 -0400

I finally got around to releasing PXSL Tools on Hackage. The package contains pxslcc, a preprocessor that converts Parsimonious XML Shorthand Language into XML, and supporting documentation.

If you want to hack on the Haskell sources, I’ve put the project on GitHub, too. See the pxsl-tools project page to browse the code, or just clone the repo and hack away:

$ git clone git://github.com/tmoertel/pxsl-tools.git

Thinking algebraically: a functional-programming dividend that pays during your imperative-programming day job

Tom Moertel — Wed, 20 Aug 2008 21:50:00 -0400

Although at work I code mostly in Python – a language from which lambda and map were nearly removed – I still find that functional-programming experience has its benefits. One of the “functional-programming dividends” I notice most often is insight gained from considering problems from an algebraic perspective.

Recently, for example, I had a small parsing problem. I had to write code that, given a simple grammar specification as input, emits a regular expression that matches the language generated by the grammar.

Here’s a simplified version of the problem. A grammar specification is limited to a series of one or more atoms. For example, “a b c” represents the atom “a”, followed by the atom “b”, followed by the atom “c”. To generate the grammar, the series of atoms is interpreted such that each atom (except the last) generates a production rule of the following form:

atom_rule ::=
  <the literal atom> (SPACE <the next rule> | NOTHING)

(SPACE represents literal white space and NOTHING represents an empty string.) The grammar as a whole is rooted in the first atom’s rule.

Thus the specification “a b c” represents the following grammar:

grammar ::= a_rule
a_rule  ::= "a" (SPACE b_rule | NOTHING)
b_rule  ::= "b" (SPACE c_rule | NOTHING)
c_rule  ::= "c"

Note that the final atom’s production matches only the literal atom itself: it has no following rule on which to chain.

The grammar, in turn, generates the following language:

a
a b
a b c

Thus, given the grammar specification “a b c”, my code had to produce a regular expression that would match “a”, “a b”, or “a b c”.

At this point, please stop for a moment and think about this little programming exercise. Do any solutions leap to mind? How would you approach the problem? Form your opinions now, because I’m going to ask you about them later. (If you’re feeling especially caffeinated, try coding a solution before reading on.)

For me, the insight that made the exercise easy was seeing that the grammar is given by folding a (suitably defined) right-associative binary operator through the series of atoms. The relationship might be easier to see if you substitute away the intermediate production rules from the grammar above:

grammar ::= "a" (SPACE "b" (SPACE "c" | NOTHING) | NOTHING)

If you squint past the SPACE and NOTHING terms, you’ll see that the grammar has the form

(a + (b + (c)))

The + is a binary operator that generates the parts we squinted away. Once you see what’s going on structurally, the operator is easy to define:

x + y = x (SPACE y | NOTHING)

Compare the operator’s definition with that of the atom_rule I presented at the beginning of the article. They’re structurally the same: the operator’s x and y are the atom rule’s <the literal atom> and <the next rule>.

Now all that remains is to generalize the “a b c” formula into a general formula that works for arbitrary grammar specifications. Fortunately, this work has already been done for us. The generalized formula is nothing more than a right fold. In Haskell, the particular right-fold flavor we want is called foldr1.

Given a list of atoms, we can use foldr1 to construct its grammar as follows:

mk_grammar atoms = foldr1 (+) atoms

But Python, our implementation language, does not offer a foldr1 function. This wrinkle, however, is another thing we can iron out by thinking algebraically. Python doesn’t have foldr1, but it does have a reduce function, which represents a left fold, equivalent to Haskell’s foldl’ or foldl1’. Because our + operator is strict and our list of atoms is finite, we can take advantage of the following identity:

foldr1 (+) xs == foldl1 (flip (+)) (reverse xs)

That is, we can convert a right fold into a left fold by flipping the arguments of the operator and operating on the list in reverse. Thus we can implement the fold we want in terms of the fold we have:

# Python code

def foldr1(f, xs):
    return reduce(flip(f), reversed(xs))

def flip(f):
    def g(x, y):
        return f(y, x)
    return g

Now writing a Python-based solution is straightforward:

def grammar_spec_to_re(spec):
    atoms = grammar_spec_to_atoms(spec)
    atom_res = map(atom_to_re, atoms)
    grammar_re = r'\A%s\Z' % foldr1(op, atom_res)
    return grammar_re

def op(x, y):
    # x + y = x (SPACE y | NOTHING)
    return r'%s(\s+%s)?' % (x, y)

def grammar_spec_to_atoms(spec):
    return spec.split()

def atom_to_re(atom):
    return re.escape(atom)

Using our solution, let’s compile the “a b c” grammar specification into its corresponding regular expression:

>>> print grammar_spec_to_re('a b c')
\Aa(\s+b(\s+c)?)?\Z

And that’s basically how I solved the problem.

To play around with the solution, here’s a small helper class that compiles a grammar specification into a regular expression and then tests strings for matching the regexp:

class GrammarMatcher(object):
    def __init__(self, spec):
        self.re = re.compile(grammar_spec_to_re(spec))
    def __call__(self, s):
        return not not self.re.match(s)

Now, let’s try out the regular expression generated for the grammar specification “a b c” :

>>> matcher = GrammarMatcher('a b c')
>>> matcher('')
False
>>> matcher('a')
True
>>> matcher('ab')
False
>>> matcher('a b')
True
>>> matcher('a  b')
True
>>> matcher('a b c')
True
>>> matcher('a b c d')
False
>>> matcher('a c')
False
>>> matcher('b')
False

Now, those questions I promised. If you’re a functional programmer, did a fold-based solution leap out at you? (Did you think of the problem in terms of folds?) If you’re not a functional programmer, how did you see the problem? Did the solution above seem twisted, confusing, or overly clever?

(There are no right or wrong answers. I’m just curious about how people with different backgrounds view the problem.)

Update: Edited to clarify that the problem is to convert a grammar specification into a regular expression, not just test whether a string matches a specified grammar.

Fun stuff: Video of Type B Erie steam shovel in action!

Tom Moertel — Sun, 10 Aug 2008 14:12:00 -0400

As an update to my previous post on the 2008 convention of the Historical Construction Equipment Association, I have posted an action-packed video of a Type-B Erie steam-powered shovel! Wait until you see this old beast belch steam and smoke and you hear it chug, clank, and huff and puff – it’s like stepping back in time. And it’s definately fun stuff!

I took this footage on 8 August 2008 in Brownsville, Pennsylvania. (I also have footage of other equipment – dozers, draglines, trucks, shovels, and more. Let me know if you’re interested, and I’ll upload those, too.)

Update: I have uploaded the rest of my footage to flickr:

Thew “O” steam shovel
Dragline
CAT 955 bulldozer
CAT D8 bulldozer
Vintage Army dump truck

See them all in my HCEA 2008 photostream.

Fun stuff: Historical Construction Equipment Association's 2008 convention

Tom Moertel — Sat, 09 Aug 2008 20:23:00 -0400

Today, my dad and I went to the 2008 annual convention of the Historical Construction Equipment Association. We were impressed with the quantity and quality of the machinery on active display: steam shovels, dozers, graders, crawlers, scrapers, cranes, steamrollers, and a bunch of other old but well-maintained construction equipment. I’m talking dozens of massive machines – not just sitting there, but working!

This year’s convention is in Brownsville, Pennsylvania and runs through August 10, 2008. If you are within driving distance and think smoke-belching, earth-shaking construction equipment is fun stuff, don’t miss it. There’s still time to go.

If you can’t make it, I took some photos for you. Not quite the real thing, but better than nothing.

Update: I have posted a video of a Type B Erie steam shovel – the one pictured above – and it’s in action! Don’t miss it.

Oleg's great way of explaining delimited continuations

Tom Moertel — Mon, 05 May 2008 09:58:00 -0400

There’s a great way to explain delimited continuations in the notes of Oleg’s Continuation Fest talk on using delimited continuations for CGI programming. Just so it doesn’t get overlooked, here it is:

I’m obsessed in pointing out that every programmer already knows and understands the delimited continuations; they might not know that word though. Everyone knows that when a process executes a system call like read, it gets suspended. When the disk delivers the data, the process is resumed. That suspension of a process is its continuation. It is delimited: it is not the check-point of the whole OS, it is the check-point of a process only, from the invocation of main() up to the point main() returns. Normally these suspensions are resumed only once, but can be zero times (exit) or twice (fork).

I especially like the final part about exit and fork, which drives home the notion that something more subtle than returning from a typical function call is going on. If anybody is confused over what suspended means, that last part ought to clear things up.

The next time I need to explain delimited continuations, I know how I’m going to do it.

That looks about right

Tom Moertel — Fri, 11 Apr 2008 11:58:00 -0400

Via Chris:

$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head
    196 git
    110 l
    102 cd
     70 make
     34 darcs
     30 pushd
     23 ssh
     23 m
     23 ls
     20 rm

The l and m commands are aliases:

l = ls –CF
m = less

Property checking with Python's nose testing framework

Tom Moertel — Wed, 19 Mar 2008 22:34:00 -0400

At work recently I was writing some tests with Python’s out-of-the-box unit-testing framework unittest. I’m new to Python and accustomed to Perl and Haskell’s testing frameworks, which are lightweight and let you write tests without much hoop-jumping. In particular, QuickCheck and LectroTest make it easy to test at the property level instead of the test-case level. With unittest, I was having to write a lot of code to get the same level of abstraction.

By “property level,” here’s what I mean. Say I’m testing this thing, let’s call it a subscriber pool. It has two fundamental properties:

Subscribe. For all initial states of the pool, if you call subscribe(user), then, assuming there have been no other operations on the pool, user must be in the pool.
Unsubscribe. For all initial states of the pool, if you call unsubscribe(user), then, assuming there have been no other operations on the pool, user must not be in the pool.

That’s it. If my implementation satisfies both properties, it’s correct. (This is a simplified version of my real testing problem, which required additional property checks.)

To test whether my implementation satisfies each property, I must write individual test cases that together “cover” the property. For example, to test whether the Subscribe property holds, I might write four test cases:

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testEmpty(self):
        load_pool_with_members([])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testOtherGuyAlreadyInPool(self):
        load_pool_with_members(["2"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAlreadyInPool(self):
        load_pool_with_members(["1"])
        subscribe("1")
        self.assert_("1" in pool_members())

    def testSubscriberAndOtherGuyAlreadyInPool(self):
        load_pool_with_members(["1", "2"])
        subscribe("1")
        self.assert_("1" in pool_members())

Every one of the test cases has the same form. The repetition makes me want to refactor the whole thing.

Okay, let’s do it:

INITIAL_POOL_STATES = [[], ["2"], ["1"], ["1", "2"]]

class SubscribeProperty(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

We’re fighting a bit with the testing framework because our notion of when set-up and tear-down should occur doesn’t match its own, but otherwise our code is looking much more manageable. In particular, if we want to extend our property-check coverage with additional initial pool states, we don’t need to write additional tests; instead, we can just extend a single list.

But we’re only halfway done. We must also check the Unsubscribe property. The code for it is virtually the same as for Subscribe, but with subscribe becoming unsubscribe and in becoming not in. Let’s add it to our class:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                subscribe("1")
                self.assert_("1" in pool_members(), case)
            finally:
                self.tearDown()

    def testUnsubscribe(self):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                unsubscribe("1")
                self.assert_("1" not in pool_members(), case)
            finally:
                self.tearDown()

And now let’s factor out the new redundancy:

class SubscriberPoolProperties(unittest.TestCase):

    def setUp(self):
        initialize_pool()

    def tearDown(self):
        destroy_pool()

    def testSubscribe(self):
        def testfn(case):
            subscribe("1")
            self.assert_("1" in pool_members(), case)
        self._forall_test_cases(testfn)

    def testUnsubscribe(self):
        def testfn(case):
            unsubscribe("1")
            self.assert_("1" not in pool_members(), case)
        self._forall_test_cases(testfn)

    def _forall_test_cases(self, testfn):
        for case in INITIAL_POOL_STATES:
            self.setUp()
            try:
                load_pool_with_members(case)
                testfn(case)
            finally:
                self.tearDown()

It’s not bad, but it’s not great either. There’s still a lot of noise in that code.

After discussing the situation with my more-Pythonic colleague Tim Lesher, I took his advice to check out the nose testing framework.

One of the things I liked right away about nose was that it supports test generators, which would let me represent each property-check as a generator that yields the test cases needed to check the property. Also, set-up and tear-down would automatically occur per generated test, so I wouldn’t have to invoke them manually.

Once I got familiar with nose, it was easy to create a decorator to represent the forall-test-cases idiom:

def forall_cases(cases):
    def decorate(testfn):
        def gen():
            for case in cases:
                yield testfn, case
        gen.__name__ = "test_%s_for_a_case" % testfn.__name__
        return gen
    return decorate

Note that this decorator is not specific to our subscriber-pool tests. It can be used in any situation where we need to check a property across a collection of cases. In fact, I keep this little gem in a “nosehelpers” library, where I reuse it all the time. Here’s an example of how to use it to check the trivial property that x = x for all x in 0–99:

@forall_cases(range(100))
def check_self_equality(x):
    assert x == x

Now, back to our testing problem. Here’s how we can use the decorator to check the Subscribe property:

@forall_cases(INITIAL_POOL_STATES)
@with_setup(initialize_pool, destroy_pool)
def check_subscribe(case):
    load_pool_with_members(case)
    subscribe("1")
    assert "1" in pool_members()

(The with_setup decorator is defined by nose and tells nose to run the given set-up and tear-down functions before and after each of the generated test cases.)

Not bad. The only problem I have with that code is that it mixes the “For all initial states of the pool” part of the property definition into the “if you call subscribe(user), then …” part. I’d like the code to be more explicit about which part defines the scope of the property claim and which part defines the test for whether the claim holds for any particular test case within that scope.

Fortunately, we can build upon our existing decorator to create exactly what we need:

def forall_initial_pools(testfn):
    @forall_cases(INITIAL_POOL_STATES)
    @with_setup(initialize_pool, destroy_pool)
    def setup_case_and_test_it(case):
        load_pool_with_members(case)
        testfn(case)
    setup_case_and_test_it.__name__ = \
        "test_%s_for_a_subscriber_case" % testfn.__name__
    return setup_case_and_test_it

Here’s what the decorator does. When you apply it to a test function testfn, it returns a test generator that yields a property-check test for each of the initial pool states. For each, it sets up a new pool, loads it with the initial subscribers (as given by the corresponding test case), runs the given check function testfn, and then cleans up after itself.

With this decorator, our Pythonic property definitions now mirror the human-language definitions from the start of the article:

@forall_initial_pools
def check_subscribe(case):
    subscribe("1")
    assert "1" in pool_members()

@forall_initial_pools
def check_unsubscribe(case):
    unsubscribe("1")
    assert "1" not in pool_members()

And that’s pretty much the solution I ended up using at work. There, as opposed to here, I got to reuse my decorators for many more tests, making them all the more worth their small implementation price.

PXSL Tools 1.0: Your ticket out of XML Hell

Tom Moertel — Mon, 17 Dec 2007 22:33:00 -0500

XML is fine for representing document-like things, but when it’s twisted to represent build recipes, configuration files, and little programming languages, it opens the gates to XML Hell. Once the gates are opened, the demons of cargo-cult thinking are loosed upon the world, where they are free to trick innocent programmers into working with grotesquely twisted XML documents – something no human mind was designed to comprehend. Ensnared, these programmers are slowly drawn into the depths of XML Hell, from which their lamentations echo across the universe.

When the demons of cargo-cult thinking come for you, don’t be ensnared! Instead, be prepared – with PXSL – the Parsimonious XML Shorthand Language (pronounced “pixel”).

What’s PXSL? It’s a luxurious, thermonuclear smoking jacket that you can slip on using a convenient preprocessor. Use it whenever you see grotesque XML on the horizon. Within PXSL’s plush (and stylish) protection, you can create all the nasty, twisted XML that may be demanded of you, but you need not descend into XML Hell to do it. Instead, you can work from the comfort of a well-stocked lounge, where clarity and conciseness are always on tap.

For example, here’s a snippet from an XSLT stylesheet, in the original XML:

<xsl:template match="/">
  <xsl:for-each select="//*/@src|//*/@href">
    <xsl:value-of select="."/>
    <xsl:text>&#10;</xsl:text>
  </xsl:for-each>
</xsl:template>

And here’s the same snippet, written in PXSL:

template /
  for-each //*/@src|//*/@href
    value-of .
    text <<&#10;>>

Isn’t that refreshing?

Why PXSL?

There are lots of XML shorthands available. (The PXSL FAQ lists about ten of them.) So why choose PXSL? Here’s why:

PXSL lets you intermix PXSL and XML syntax in one document. Feel free to use whichever syntax works best for each portion of your documents. (See the PXSL documentation for some examples.)
PXSL is customizable with application-specific shortcuts. (The PXSL snippet above, for example, uses XSLT shortcuts. Again, the PXSL documentation has examples.)
PXSL has a powerful macro system that lets you build complicated document structures safely and conveniently. (Read about macros in the PXSL documentation. For an advanced example of what you can do with PXSL macros, see this article on refactoring XSLT one-offs into clean, maintainable code.)

Also, PXSL is battle tested. It was first released in 2003 and has been saving people from XML Hell since. People who try it seem to like it:

I think PXSL could do wonders for soothing my irrational hatred for all things XML. —kowey
Impressive… I converted some of my files from XML to PXSL and the readability was much improved. —chris
Quite aside from the fact that XSLT is finally somewhat readable, the fact that you’ve added a serious macro system means that some serious scripting of XML can occur. I’m very impressed. —invisible

The next time you’re headed for XML Hell, why not give the venerable PXSL a try? You might just find that you like it, too.

This public service announcement was brought to you in celebration of the 1.0 release of the pxsl-tools package. The PXSL-to-XML compiler pxslcc is written in Haskell and uses the cross-platform Haskell Cabal build/package system to let you use PXSL just about anywhere.