Tom Moertel's Weblog: Tag haskell

PXSL Tools now on Hackage and GitHub

Tom Moertel — Sun, 24 Aug 2008 22:56:00 -0400

I finally got around to releasing PXSL Tools on Hackage. The package contains pxslcc, a preprocessor that converts Parsimonious XML Shorthand Language into XML, and supporting documentation.

If you want to hack on the Haskell sources, I’ve put the project on GitHub, too. See the pxsl-tools project page to browse the code, or just clone the repo and hack away:

$ git clone git://github.com/tmoertel/pxsl-tools.git

Thinking algebraically: a functional-programming dividend that pays during your imperative-programming day job

Tom Moertel — Wed, 20 Aug 2008 21:50:00 -0400

Although at work I code mostly in Python – a language from which lambda and map were nearly removed – I still find that functional-programming experience has its benefits. One of the “functional-programming dividends” I notice most often is insight gained from considering problems from an algebraic perspective.

Recently, for example, I had a small parsing problem. I had to write code that, given a simple grammar specification as input, emits a regular expression that matches the language generated by the grammar.

Here’s a simplified version of the problem. A grammar specification is limited to a series of one or more atoms. For example, “a b c” represents the atom “a”, followed by the atom “b”, followed by the atom “c”. To generate the grammar, the series of atoms is interpreted such that each atom (except the last) generates a production rule of the following form:

atom_rule ::=
  <the literal atom> (SPACE <the next rule> | NOTHING)

(SPACE represents literal white space and NOTHING represents an empty string.) The grammar as a whole is rooted in the first atom’s rule.

Thus the specification “a b c” represents the following grammar:

grammar ::= a_rule
a_rule  ::= "a" (SPACE b_rule | NOTHING)
b_rule  ::= "b" (SPACE c_rule | NOTHING)
c_rule  ::= "c"

Note that the final atom’s production matches only the literal atom itself: it has no following rule on which to chain.

The grammar, in turn, generates the following language:

a
a b
a b c

Thus, given the grammar specification “a b c”, my code had to produce a regular expression that would match “a”, “a b”, or “a b c”.

At this point, please stop for a moment and think about this little programming exercise. Do any solutions leap to mind? How would you approach the problem? Form your opinions now, because I’m going to ask you about them later. (If you’re feeling especially caffeinated, try coding a solution before reading on.)

For me, the insight that made the exercise easy was seeing that the grammar is given by folding a (suitably defined) right-associative binary operator through the series of atoms. The relationship might be easier to see if you substitute away the intermediate production rules from the grammar above:

grammar ::= "a" (SPACE "b" (SPACE "c" | NOTHING) | NOTHING)

If you squint past the SPACE and NOTHING terms, you’ll see that the grammar has the form

(a + (b + (c)))

The + is a binary operator that generates the parts we squinted away. Once you see what’s going on structurally, the operator is easy to define:

x + y = x (SPACE y | NOTHING)

Compare the operator’s definition with that of the atom_rule I presented at the beginning of the article. They’re structurally the same: the operator’s x and y are the atom rule’s <the literal atom> and <the next rule>.

Now all that remains is to generalize the “a b c” formula into a general formula that works for arbitrary grammar specifications. Fortunately, this work has already been done for us. The generalized formula is nothing more than a right fold. In Haskell, the particular right-fold flavor we want is called foldr1.

Given a list of atoms, we can use foldr1 to construct its grammar as follows:

mk_grammar atoms = foldr1 (+) atoms

But Python, our implementation language, does not offer a foldr1 function. This wrinkle, however, is another thing we can iron out by thinking algebraically. Python doesn’t have foldr1, but it does have a reduce function, which represents a left fold, equivalent to Haskell’s foldl’ or foldl1’. Because our + operator is strict and our list of atoms is finite, we can take advantage of the following identity:

foldr1 (+) xs == foldl1 (flip (+)) (reverse xs)

That is, we can convert a right fold into a left fold by flipping the arguments of the operator and operating on the list in reverse. Thus we can implement the fold we want in terms of the fold we have:

# Python code

def foldr1(f, xs):
    return reduce(flip(f), reversed(xs))

def flip(f):
    def g(x, y):
        return f(y, x)
    return g

Now writing a Python-based solution is straightforward:

def grammar_spec_to_re(spec):
    atoms = grammar_spec_to_atoms(spec)
    atom_res = map(atom_to_re, atoms)
    grammar_re = r'\A%s\Z' % foldr1(op, atom_res)
    return grammar_re

def op(x, y):
    # x + y = x (SPACE y | NOTHING)
    return r'%s(\s+%s)?' % (x, y)

def grammar_spec_to_atoms(spec):
    return spec.split()

def atom_to_re(atom):
    return re.escape(atom)

Using our solution, let’s compile the “a b c” grammar specification into its corresponding regular expression:

>>> print grammar_spec_to_re('a b c')
\Aa(\s+b(\s+c)?)?\Z

And that’s basically how I solved the problem.

To play around with the solution, here’s a small helper class that compiles a grammar specification into a regular expression and then tests strings for matching the regexp:

class GrammarMatcher(object):
    def __init__(self, spec):
        self.re = re.compile(grammar_spec_to_re(spec))
    def __call__(self, s):
        return not not self.re.match(s)

Now, let’s try out the regular expression generated for the grammar specification “a b c” :

>>> matcher = GrammarMatcher('a b c')
>>> matcher('')
False
>>> matcher('a')
True
>>> matcher('ab')
False
>>> matcher('a b')
True
>>> matcher('a  b')
True
>>> matcher('a b c')
True
>>> matcher('a b c d')
False
>>> matcher('a c')
False
>>> matcher('b')
False

Now, those questions I promised. If you’re a functional programmer, did a fold-based solution leap out at you? (Did you think of the problem in terms of folds?) If you’re not a functional programmer, how did you see the problem? Did the solution above seem twisted, confusing, or overly clever?

(There are no right or wrong answers. I’m just curious about how people with different backgrounds view the problem.)

Update: Edited to clarify that the problem is to convert a grammar specification into a regular expression, not just test whether a string matches a specified grammar.

PXSL Tools 1.0: Your ticket out of XML Hell

Tom Moertel — Mon, 17 Dec 2007 22:33:00 -0500

XML is fine for representing document-like things, but when it’s twisted to represent build recipes, configuration files, and little programming languages, it opens the gates to XML Hell. Once the gates are opened, the demons of cargo-cult thinking are loosed upon the world, where they are free to trick innocent programmers into working with grotesquely twisted XML documents – something no human mind was designed to comprehend. Ensnared, these programmers are slowly drawn into the depths of XML Hell, from which their lamentations echo across the universe.

When the demons of cargo-cult thinking come for you, don’t be ensnared! Instead, be prepared – with PXSL – the Parsimonious XML Shorthand Language (pronounced “pixel”).

What’s PXSL? It’s a luxurious, thermonuclear smoking jacket that you can slip on using a convenient preprocessor. Use it whenever you see grotesque XML on the horizon. Within PXSL’s plush (and stylish) protection, you can create all the nasty, twisted XML that may be demanded of you, but you need not descend into XML Hell to do it. Instead, you can work from the comfort of a well-stocked lounge, where clarity and conciseness are always on tap.

For example, here’s a snippet from an XSLT stylesheet, in the original XML:

<xsl:template match="/">
  <xsl:for-each select="//*/@src|//*/@href">
    <xsl:value-of select="."/>
    <xsl:text>&#10;</xsl:text>
  </xsl:for-each>
</xsl:template>

And here’s the same snippet, written in PXSL:

template /
  for-each //*/@src|//*/@href
    value-of .
    text <<&#10;>>

Isn’t that refreshing?

Why PXSL?

There are lots of XML shorthands available. (The PXSL FAQ lists about ten of them.) So why choose PXSL? Here’s why:

PXSL lets you intermix PXSL and XML syntax in one document. Feel free to use whichever syntax works best for each portion of your documents. (See the PXSL documentation for some examples.)
PXSL is customizable with application-specific shortcuts. (The PXSL snippet above, for example, uses XSLT shortcuts. Again, the PXSL documentation has examples.)
PXSL has a powerful macro system that lets you build complicated document structures safely and conveniently. (Read about macros in the PXSL documentation. For an advanced example of what you can do with PXSL macros, see this article on refactoring XSLT one-offs into clean, maintainable code.)

Also, PXSL is battle tested. It was first released in 2003 and has been saving people from XML Hell since. People who try it seem to like it:

I think PXSL could do wonders for soothing my irrational hatred for all things XML. —kowey
Impressive… I converted some of my files from XML to PXSL and the readability was much improved. —chris
Quite aside from the fact that XSLT is finally somewhat readable, the fact that you’ve added a serious macro system means that some serious scripting of XML can occur. I’m very impressed. —invisible

The next time you’re headed for XML Hell, why not give the venerable PXSL a try? You might just find that you like it, too.

This public service announcement was brought to you in celebration of the 1.0 release of the pxsl-tools package. The PXSL-to-XML compiler pxslcc is written in Haskell and uses the cross-platform Haskell Cabal build/package system to let you use PXSL just about anywhere.

How I stopped missing Darcs and started loving Git

Tom Moertel — Mon, 10 Dec 2007 16:52:00 -0500

About three years ago, I switched to Darcs as my primary source-code management system. It was simple, intuitive, and powerful, and it made managing my projects more fun and less frustrating than any centralized VCS ever had. That it was written in Haskell, one of my favorite programming languages, made it even better. I was hooked.

Since then, the distributed SCM landscape has changed. Darcs hasn’t improved much, but its competitors have made long strides, especially Git and Mercurial. Both are crazy fast, vigorously developed, and widely used on large, highly active real-world projects, such as the Linux kernel and Mozilla 2. In comparison, Darcs has stagnated.

When I started working for a new company recently, I had to consider whether to advocate Darcs or something else. In the end, I decided that Darcs would be a hard sell. Nobody else at the company uses Haskell, and having to explain how to avoid the occasional corner case seemed liked a losing proposition.

After researching and playing around with Git and Mercurial, I settled on Git. I like Git’s underlying hashed-blobs model better than Mercurial’s revlogs, and Git seems to have slightly more development momentum. Still, it was a close call. Either choice would have been completely reasonable.

Missing Darcs

When I started using Git on real projects, the one thing I really missed was the ability to easily amend earlier patches, something Darcs made trivial. Let me explain. The typical development workflow goes something like this:

Checkout copy of upstream code base.
Implement feature X.
Commit.
Implement independent feature Y.
Commit.
Implement independent feature Z.
Commit.
Push new features back upstream.

Now, what really happens is that when I’m implementing Y or Z, I’ll realize that I made a mistake in X. The trick is then fixing X so that my fix is part of the changeset/patch for X that ultimately gets pushed upstream in the last step. That way, the upstream folks will see only a single, clean patch for feature X – not a mishmash of patches that together represent X.

In Darcs, amending the original patch is easy because its patch theory lets me tweak the patch for X independently of the other patches. Darcs will simply ask me which patch I want to amend, and I’ll select the orignal patch for X:

$ emacs               # fix X
$ darcs amend-record  # amend original patch for X

Mon Dec 10 14:43:13 EST 2007  Tom Moertel <tom@moertel.com>
  * Implemented Z
Shall I amend this patch? [yNvpq], or ? for help: n

Mon Dec 10 14:42:12 EST 2007  Tom Moertel <tom@moertel.com>
  * Implemented Y
Shall I amend this patch? [yNvpq], or ? for help: n

Mon Dec 10 14:41:46 EST 2007  Tom Moertel <tom@moertel.com>
  * Implemented X
Shall I amend this patch? [yNvpq], or ? for help: y
hunk ./x 1
-X1
+X2
Shall I add this change? (1/?)  [ynWsfqadjkc], or ? for help: y
Finished amending patch:
Mon Dec 10 14:43:25 EST 2007  Tom Moertel <tom@moertel.com>
  * Implemented X

That’s it. The exact same process will work regardless of when I realize I need to fix X: before I start Y, while I’m implementing Y, after I’ve committed Y, while I’m working on Z, or after I’ve committed Z.

Learning to love Git

With Git, however, I can amend a commit only if I haven’t committed anything else before making my fix. In Git’s mind, Y depends on X, and Z depends on Y, even if they really are independent of one another.

So if I commit the original patch for X and then immediately realize I need to make a fix, before I start working on Y or Z, it’s easy:

$ emacs               # implement X
$ git commit -m 'Implemented X'

# discover problem in X

$ emacs               # fix X
$ git commit --amend  # amend original patch

More typically, it’s only while I’m working on Y that I’ll realize I need to fix X. Then it’s more complicated to amend the original commit:

$ emacs               # implement X
$ git commit -m 'Implemented X'
$ emacs               # start working on Y

# discover problem in X

$ git stash           # stash away half-completed work on Y
$ emacs               # fix X
$ git commit --amend  # amend original patch for X
$ git stash apply     # restore work on Y
$ emacs               # continue working on Y

While not as convenient as Darcs’s workflow, it’s perfectly workable.

Now let’s consider another fairly typical case: I commit X and Y and then start working on Z before I notice the problem in X. I used to think that Git couldn’t handle this case, but it can, thanks to git rebase --interactive:

$ emacs               # implement X
$ git commit -m 'Implemented X'
$ emacs               # implement Y
$ git commit -m 'Implemented Y'
$ emacs               # start working on Z

# discover problem in X

$ git stash           # stash away half-completed work on Z
$ emacs               # fix X
$ git commit -m 'Fixed X'
$ git rebase --interactive HEAD~3  # see comments below
$ git stash apply     # restore work on Z
$ emacs               # continue working on Z

The git rebase --interactive command is powerful. What the command does, as called in the snippet above, is invoke my editor of choice on a text file describing the last 3 commits (that’s the HEAD~3 part):

# Rebasing 3ad99a7..b9a8405 onto 3ad99a7
#
# Commands:
#  pick = use commit
#  edit = use commit, but stop for amending
#  squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
pick 0885540 Implemented X
pick 320b115 Implemented Y
pick b9a8405 Fixed X

I can then edit the file to reorder, merge (squash), and/or remove the commits. In this example, I want to merge the fix for X into the original commit that implemented X. So I edit the file like so:

pick 0885540 Implemented X
squash b9a8405 Fixed X
pick 320b115 Implemented Y

Then I save the file, at which point Git takes over and makes the requested changes, merging the fix for X into the original commit for X. Now the log shows the original implementation and fix as one commit:

$ git log
commit f387d650976246c0854d028b040cca40e542be56
Author: Tom Moertel <tom@moertel.com>
Date:   Mon Dec 10 15:11:26 2007 -0500

    Implemented Y

commit 82a1c849ffd1bd688d5bc9d99be0e63548a89c4c
Author: Tom Moertel <tom@moertel.com>
Date:   Mon Dec 10 15:13:03 2007 -0500

    Implemented X

    Fixed X

commit 3ad99a7ef537b7ae99e435e0d2b4b0d03de92c65
Author: Tom Moertel <tom@moertel.com>
Date:   Mon Dec 10 15:11:14 2007 -0500

    Initial checkin

Once I figured out how to use git rebase --interactive, I stopped missing Darcs and started loving Git.

TMR 9!

Tom Moertel — Mon, 19 Nov 2007 13:07:00 -0500

Issue 9 of The Monad.Reader is hot off the presses! The issue focuses on three Google-Summer-of-Code projects for Haskell: Cabal configurations, Darcs’s Patch Theory, and the typechecker-framework TaiChi. Good stuff.

I know what I’ll be reading for lunch today.

Updated cabal2rpm helps you make RPM packages from Haskell Cabal packages

Tom Moertel — Fri, 07 Sep 2007 20:19:00 -0400

I just released an updated version of cabal2rpm, a small program (written in Perl) that creates RPM spec files from Cabal package descriptions. RPM is the software-packaging format used by several popular Linux distributions, including Red Hat and Fedora. Cabal is the packaging format used by the Haskell community to distribute software written in Haskell.

Bryan O’Sullivan’s cabal-rpm also creates spec files from Cabal packages. Unlike cabal2rpm, it is written in Haskell and directly interfaces with the Cabal libraries. Long term, it is the way to go. For now, however, cabal2rpm may be more convenient because it works out of the box. (To use cabal-rpm, you’ll first need to install the just-tagged Cabal 1.2.0 library, not yet in wide distribution.)

ClusterBy: a handy little function for the toolbox

Tom Moertel — Sat, 01 Sep 2007 15:39:00 -0400

Via Reddit I found Mark Nelson’s post about a recent word puzzle from NPR’s Weekend Edition:

Take the names of two U.S. States, mix them all together, then rearrange the letters to form the names of two other U.S. States. What states are these?

The puzzle is fairly straightforward to solve by hand (think about it), but let’s write a program to solve it. That will give us a convenient excuse to discuss a super-handy function I use all the time: clusterBy. In Haskell, it looks like this:

import Control.Arrow ((&&&))
import qualified Data.Map as M

clusterBy :: Ord b => (a -> b) -> [a] -> [[a]]
clusterBy f = M.elems . M.map reverse . M.fromListWith (++)
            . map (f &&& return)

What clusterBy does is group a list of values by their signatures, as computed by a given signature function f, and returns the groups in order of ascending signature. For example, we can cluster the words “the tan ant gets some fat” by length, by first letter, or by last letter just by changing the signature function we give to clusterBy:

*Main> let antwords = words "the tan ant gets some fat"

*Main> clusterBy length antwords
[["the","tan","ant","fat"],["gets","some"]]

*Main> clusterBy head antwords
[["ant"],["fat"],["gets"],["some"],["the","tan"]]

*Main> clusterBy last antwords
[["the","some"],["tan"],["gets"],["ant","fat"]]

If we use sort as the signature function, we can find anagrams:

*Main> clusterBy sort antwords
[["fat"],["tan","ant"],["gets"],["the"],["some"]]

And that brings us back to the original puzzle. To find the solution, we must consider each unique pair of state names to form a “word” and find the anagrams among a list of such “words.”

Assuming we are given a list of state names on standard input, one state per line, we can write the shell of our solution as follows:

main = mapM_ print . solve . lines =<< getContents

The shell delegates the real work to solve. It’s job is to compute the unique, 2-state combinations from the original list of states, and then find the anagrams among these combinations. As before, finding the anagrams is simply a matter of calling clusterBy with the right signature function. We also filter out the trivial results, which are not valid solutions:

solve = filter ((>1) . length) . clusterBy signature . ucombos
ucombos xs = [[x,y] | x <- xs, y <- xs, x < y]
signature = sort . filter isAlpha . concat   -- sort letters

That’s it. Now we can solve the puzzle by feeding our program a list of states:

$ runhaskell anagrams2.hs < states.txt

[["NORTH CAROLINA","SOUTH DAKOTA"],
 ["NORTH DAKOTA","SOUTH CAROLINA"]]

What a handy little function, that clusterBy.

Update: made clear that clusterBy returns clusters in order of ascending signature.

Update 2007-10-31: For more interesting discussion of clusterBy and the original puzzle from NPR, see Anders Pearson’s blog: A Simple Programming Puzzle Seen Through Three Different Lenses.

Directory-tree printing in Haskell, part three: lazy I/O

Tom Moertel — Wed, 28 Mar 2007 15:40:00 -0400

This article is part three in a series on introductory Haskell programming. In the first article, we wrote a small program to recursively scan file-system directories and print their contents as ASCII-art trees. In the second article, we refactored the program to make its logic more reusable by separating the directory-scanning logic from the tree-printing logic. In this article, we will address a shortcoming of the refactored version: It must scan directory hierarchies completely before printing their trees, i.e., it must scan and then print, when doing both simultaneously is both more efficient and more user friendly.

Recall from the previous article that our directory-printing program is factored into three pieces of logic:

fsTraverse, which traverses a file-system hierarchy and returns a tree data structure;
showTree, which converts a tree into lovingly crafted ASCII art; and
traverseAndPrint, which prints the tree for a file-system hierarchy by using the first two pieces of logic.

The types of the functions are as follows:

fsTraverse       :: Path -> DentName -> IO DirTree
showTree         :: Tree String -> String
traverseAndPrint :: Path -> IO ()

Note that showTree is a pure function, but the other two return IO actions that may have side effects.

Within traverseAndPrint, fsTraverse and showTree are combined into a composite IO action by the =<< combinator:

putStr . showTree =<< fsTraverse root path

The sequencing semantics of Haskell’s IO monad forces all of the effects of fsTraverse to complete before any following effects can begin. To better understand these sequencing semantics, let’s consider a simple example.

The IO-monad code,

a >> b

can loosely be interpreted as running the action a, which forces its side effects to occur, and then running the action b, which forces its side effects to occur.

In reality, a and b are not actions. They are functions. Like all Haskell functions, they are pure and have no side effects. It’s just that a and b return values that represent actions, and those actions may have side effects, and the semantics of the IO monad guarantee the ordering of those effects (should the actions end up being connected to the runtime’s top-level IO action and executed). If you think that’s weird, hold that thought. For now, all that’s important is that, if the composite action represented by the expression (a >> b) is executed, the effects of a, regardless of how complex, will be executed before the effects of b.

Thus if a represents building a tree by recursively scanning a file-system hierarchy, the entire tree must be built before b ever gets a chance to do its thing. For our particular application, however, that particular sequencing is suboptimal. We know from our earlier, monolithic implementation that the file-system hierarchy can be scanned and printed simultaneously, which is more efficient. Ideally, then, our refactored implementation should be just as efficient.

In this article, we will look at one way to maintain the clean, logical separation of the a part from the b part while allowing the parts’ effects to be interleaved for efficiency. We will use an extension to the Haskell language to make the directory-scanning action lazy so that it builds the tree as the tree is consumed.

Ready? Let’s dive in.

Lazy I/O

I’ll be up front: we are going to be naughty. We are going to break the effect-ordering guarantees of the IO monad, but, if we are careful, we can get away with it. In fact, if you have used Haskell’s getContents or readFile functions, you have already used the fruits of this very naughtiness.

Before we go down the dark path, however, let’s spend a moment understanding why we might want to be naughty. Currently, our tree-building code builds the entire tree for a file-system hierarchy as one large IO action. If the hierarchy turns out to be large, the tree could gobble up lots of memory before the printing process has the chance to consume it. Further, if the underlying file-system operations are expensive, we could wait a long time before seeing any output from our program.

If we could produce the tree lazily, both of these problems would go away. As we built the tree, our tree-printing process would consume it, keeping the overall memory footprint low and keeping us informed of progress with timely output.

To better visualize our tree-building code’s laziness or lack thereof, let’s artificially raise the cost of visiting a file-system node. We will introduce fsVisit to simulate a costly file-system operation that takes 0.1 second to complete:

fsVisit :: DirNode -> IO DentName
fsVisit (_, name) = do
    usleep 100000
    return name

Now let’s work fsVisit into each step of our file-system-traversal logic by revising fsTraverseStep:

fsTraverseStep :: DirNode -> IO (DentName, [DirNode])
fsTraverseStep dnode@(path, node) = do
    name     <- fsVisit dnode
    children <- fsGetChildren (path ++ "/" ++ node)
    return (name, children)

Now have a slow version of our original tlist program. I recommend that you download the source code and compile it so that you can run the program locally and experience the timing of its output.

If you run the program on a directory that contains a handful of files, it will pause for a second or two while it performs our artificially expensive directory scan. Finally, the scan completed, the program will print the directory tree all at once.

Now let’s make the tree-building process lazy. For this, we need the unsafeInterleaveIO combinator, an extension to Haskell 98. (We will examine why it is “unsafe” later.) Here is its type signature:

unsafeInterleaveIO :: IO a -> IO a

What the combinator does is convert an action into a promise to perform the action later, should its result be needed. It takes an IO action, which might take a long time to carry out, and returns a substitute action that produces a value immediately. That value, however, is a placeholder: when you try to consume it, you will be made to wait while the real value is computed by performing the original action.

This ability to defer work via promises is exactly what we need to make our directory scans lazy. We can return each node of the tree as a pair of promises, one to compute the node’s root, and one to compute the node’s children. Let’s modify fsTraverseStep to do just that:

import System.IO.Unsafe -- add to imports at top of code

fsTraverseStep dnode@(path, node) = do
    name     <- unsafeInterleaveIO $ fsVisit dnode
    children <- unsafeInterleaveIO $
                fsGetChildren (path ++ "/" ++ node)
    return (name, children)

Now run this modified version of the program. (For convenience, you can download this version of the source code.) Notice how each line of the directory tree is printed as it is visited in the file-system scan. We have made our tree-building process lazy!

Or have we? Let’s modify fsGetChildren to print the list of visible children as they are discovered:

fsGetChildren path = do
    contents <- getDirectoryContents path `catch` const (return [])
    let visibles = sort . filter (`notElem` [".", ".."]) $ contents
    print visibles  -- <--- add this line
    return (map ((,) path) visibles)

Recompiling and re-running our “lazy” program shows that all of the children are found before any of the printing occurs. In other words, even if we are visiting each node lazily, we are still finding all of the nodes in the hierarchy before the first node of the resulting tree is emitted from fsTraverse.

To see why, let’s examine the source code for unfoldTreeM, which is the library function we used to build the tree using our fsTraverseStep function:

unfoldTreeM :: (b -> IO (a, [b])) -> b -> IO (Tree a)
unfoldTreeM step seed = do
    (root, seeds) <- step seed                -- (1)
    children <- mapM (unfoldTreeM step) seeds -- (2)
    return (Node root children)               -- (3)

When unfoldTreeM calls our step function in line (1), it gets back a pair of promises, one for the root of the tree and another for the seeds from which to build the root’s children. In line (2), we see the problem: In order to convert seeds into a list of child nodes, unfoldTreeM calls mapM to recursively apply itself to seeds. The mapM action is sequenced before the return in line (3), and thus the value of seeds is consumed before the new tree Node can be constructed. In other words, because the mapM action isn’t deferred and because it depends on the value of seeds, it forces us to make good on our seeds promise earlier than we would like.

To get around this problem, we can create a lazy variant of unfoldTreeM. This version defers the mapM action until the children value is consumed:

lazyUnfoldTreeM :: (b -> IO (a, [b])) -> b -> IO (Tree a)
lazyUnfoldTreeM step seed = do
    (root, seeds) <- step seed
    children <- unsafeInterleaveIO $
                mapM (lazyUnfoldTreeM step) seeds
    return (Node root children)

Then we can use this version in our high-level fsTraverse function:

fsTraverse = curry (lazyUnfoldTreeM fsTraverseStep)

These changes made, we have a fully lazy version of the tree-building code. To see it in action, download this version of the whole program’s source code, compile it, and give it a try. If you add the print line to fsGetChildren like we did earlier, you’ll see that this version expands each directory only when its contents are needed. Now our lazy directory-tree-building code works as we would expect.

Mission accomplished.

Optional: Hiding the plumbing

[Note: The technique described in this section is hacky. I’m including it because it’s fun, not because I think it’s good Haskell-programming practice.]

It’s unfortunate that we had to create our own version of the library function unfoldTreeM. With its introduction, we lost some of our code’s tidiness. Can we get our clean code back?

Maybe we can. Recall that unfoldTreeM works for any monad. What if we created a LazyIO monad that was just like IO except that each action within it was performed only when its value was consumed? With this monad, we could eliminate the need for a special, lazy version of unfoldTreeM: the normal version would “do the right thing,” using our monad.

Let’s create our LazyIO monad. First, we’ll need a new module,

module LazyIO
 ( LazyIO(), runLazy, deferIO )
where

and, of course, we’ll need to import unsafeInterleaveIO:

import System.IO.Unsafe (unsafeInterleaveIO)

We will define LazyIO to be a type wrapper around the regular IO type:

newtype LazyIO a = LazyIO { unLazyIO :: IO a }

Note that our module definition, above, does not export the LazyIO type’s data constructor. The only way, then, to construct and destruct values will be through our interface.

To construct a value, we will provide deferIO, which converts a normal IO action into a LazyIO action:

deferIO :: IO a -> LazyIO a
deferIO = LazyIO

To get a regular IO action “out” of our monad, we will provide runLazy which converts a LazyIO action into an IO action that is deferred until it is consumed:

runLazy :: LazyIO a -> IO a
runLazy = unsafeInterleaveIO . unLazyIO

The key to our monad is that the only way to get an IO action out of LazyIO is through runLazy and thus through unsafeInterleaveIO. In fact, injecting an IO action into our monad and then immediately pulling it out again is identical to deferring it with unsafeInterleaveIO:

(runLazy . deferIO) = unsafeInterleaveIO

But what makes a monad interesting are its return and bind definitions. Here are ours:

instance Monad LazyIO where
    return     = LazyIO . return
    ma >>= fmb = LazyIO $ runLazy ma >>= unLazyIO . fmb

Our return simply returns the given value in the underlying IO monad and wraps the resulting action with the LazyIO constructor. The implication is that the only way to unwrap the action is via runLazy, which will defer it.

Likewise, our bind (>>=) creates a composite LazyIO action from another LazyIO action and a function that produces a LazyIO action given the result of the first action. The composite action and the first action are deferred. (Deferring the action produced by fmb would be redundant because the value it produces is the value of the composite action, which is already deferred.)

Exercise: Is LazyIO really a monad? Prove (or disprove) that it satisfies the monad laws.

To see the effect of our monad, let’s apply its definitions to the following simple lazy IO action.

runLazy $ do
    x <- return 5
    deferIO (print x)

With a little expansion, application, and simplification (try it), we arrive at the following:

unsafeInterleaveIO $ do
    x <- unsafeInterleaveIO (return 5)
    print x

Note that the entire, composite IO action is deferred, as are the actions upon which it depends, which is exactly what we want.

Now we can return to housecleaning. With our new monad, we can banish both unsafeInterleaveIO and our custom, lazy version of unfoldTreeM from our otherwise uncluttered code.

Step one of the housecleaning is to modify fsTraverseStep to use the LazyIO monad instead of explicit calls to unsafeInterleaveIO:

import LazyIO   -- add to module imports at top of code

fsTraverseStep :: DirNode -> LazyIO (DentName, [DirNode])
fsTraverseStep dnode@(path, node) = do
    name     <- deferIO (fsVisit dnode)
    children <- deferIO (fsGetChildren (path ++ "/" ++ node))
    return (name, children)

Step two is to rewrite fsTraverse in terms of runLazy and the plain-old unfoldTreeM provided by the Data.Tree library:

fsTraverse :: Path -> DentName -> IO DirTree
fsTraverse p n = runLazy (unfoldTreeM fsTraverseStep (p,n))

Step three is to remove lazyUnfoldTreeM from our code: Snip!

And we’re done. The code:

To compile, here’s the command to use:

ghc -O2 -o tlist-slow-lazy-final --make tlist-slow-lazy-final.hs

Give the program a test run and convince yourself that it works as advertised.

Why unsafeInterleaveIO is “unsafe”

Now let’s get back to the “unsafe” part of unsafeInterleaveIO. What’s that about?

Haskell programmers label as “unsafe” those functions that allow you to circumvent Haskell’s safety guarantees. It’s a blunt way of letting you know that, if you use one of these functions, you can shoot yourself in the foot if your aim is bad. Thus, by using an “unsafe” function, you place the burden of proving that your code is safe upon yourself.

As we have already seen, we can use unsafeInterleaveIO to get around the side-effect ordering guarantees that Haskell’s IO monad normally provides. If we’re not careful, this could come back to bite us.

Using unsafeInterleaveIO on an action effectively removes the action from the normal world of IO actions and places it into its own parallel world, where it can run on its own schedule. Creating these parallel worlds is safe as long as we don’t attempt to do anything that depends upon the ordering of effects across worlds.

For example, take a look at this small Haskell program that will let us compare the output of a simple test under normal and unsafe-interleaved I/O sequencing:

{- file: unsafe-demo.hs -}

module Main where

import System.Environment
import System.IO.Unsafe

main = do
    args <- getArgs
    if args == ["lazy"]
        then doTest (unsafeInterleaveIO getLine)
        else doTest getLine

doTest myGetLine = do
    line1 <- myGetLine    -- (1)
    line2 <- myGetLine    -- (2)
    putStrLn line2
    putStrLn line1

The main function checks the command line to see if we have passed the “lazy” argument. If so, it passes (unsafeInterleaveIO getLine) to doTest; otherwise, it passes the straight getLine function.

The doTest function, in turn, constructs an IO action that reads two lines from the standard input using whichever version of getLine we have provided. First it reads line1, then line2. Finally, it prints the lines in the opposite order: first line2, then line1.

Let’s test this program on the following two-line file:

$ cat onetwo.txt
one
two

Ready? First, let’s run the straight-IO version:

$ runhaskell unsafe-demo.hs < onetwo.txt
two
one

As we would expect, the program read the lines from the file and printed them in reverse order.

Now, let’s pass the “lazy” flag, which wraps the getLine calls with unsafeInterleaveIO:

$ runhaskell unsafe-demo.hs lazy < onetwo.txt
one
two

For some reason, when we printed line2 we got the first line of the file, not the second. Likewise, when we printed line1, we got the second line. What happened?

What happened is that in lines (1) and (2) of the code, myGetLine evaluated to (unsafeInterleaveIO getLine). Thus line1 and line2 were bound to placeholder values that were connected to deferred getLine actions, each existing in its own parallel world and utterly unconnected to each other or to the main, normal IO world. The only thing, then, that could determine the order in which the two actions were performed was the order in which their values were consumed. And, because we printed line2 first, its deferred getLine action was the first to be performed, and so it read the first line of the input from the file. By the time we printed line1, causing its deferred getLine action to be performed, the first line of the file had already been consumed, and so the second line was returned.

Oops.

After this demonstration, we might wonder whether our lazy tree-building code is safe. After all, we’re using unsafeInterleaveIO like crazy behind the scenes. Let’s think about it.

Recall that, once detached from the main IO world, the only thing that determines when a deferred action is performed is when its value is consumed. That knowledge gives us the key to unlock the safety argument for our tree-building code.

Recall the type of fsTraverse:

fsTraverse :: Path -> DentName -> IO DirTree

The action that fsTraverse produces returns a DirTree value, which is a type synonym for a Tree of DentName values (file names). The Tree data type has only one constructor:

data Tree a = Node { rootLabel :: a,
                     subForest :: [Tree a] }

Each Node represents a tree: a root label and a sub-forest of children trees. Because of this representation, we cannot visit a node in a tree without first having visited all of its ancestors. In order to get to a child node in the tree, for example, we must go through the subForest value of its parent. Attempting to consume that value will force the attached, deferred action to be performed. That action, in turn, will fetch the contents of the parent’s directory and package each item, including the child we’re trying to get, into its own Node, the root and subForest of which will contain deferred actions for scanning further into the file-system hierarchy. Thus a top-down sequencing will be imposed on the deferred IO actions attached to those nodes. Fortunately, we want a top-down traversal because that’s the way we must descend directory hierarchies in the file system. Therefore, we have good reason to be assured of the safety of using unsafeInterleaveIO for this particular application.

For other applications, however, we must repeat this safety analysis anew, lest we allow the parallel words of our deferred actions to collide in unpleasant ways. Caveat coder.

Review and next time

In this installment of our ever-lengthening series of articles on directory printing, we revised our refactored directory-printing code to restore the efficiency of our original, monolithic implementation. In doing so, we experimented with lazy I/O and deferring IO actions by using the unsafeInterleaveIO function.

At first we inserted a customized version of the library function unfoldTreeM into our code to effect the desired laziness, but a little monad hacking allowed us to get rid of this intrusion. When we were done, we not only had a lean, lazy directory printer but also a new (and potentially unsafe) LazyIO library, which might prove useful in the future.

We also looked at the kind of problems that unsafeInterleaveIO can cause if we are not careful. Nevertheless, we were able to argue that our particular use of this function is safe: the Tree data structure ensures that we must consume tree nodes in the top-down order in which our directory-scanning must occur.

Still, our directory-printing code has problems. We haven’t yet improved its lackluster error-handling policy. Nor have we made the directory-scanning code amenable to uses beyond our immediate needs for directory printing. Next time, we will solve both of these problems.