Closures and the professional programmer
Posted by Tom Moertel Tue, 30 Aug 2005 18:56:00 GMT
I came across Tim Bray’s thoughts on Ruby via the ever-delightful Lambda the Ultimate and found the following bit fascinating:
I’ve had access to languages with closures and continuations and suchlike constructs for years and years, and I’ve never ever written one. While I’m impressed by how natural this stuff is in Ruby, I’m still unconvinced that these are a necessary part of the professional programmer’s arsenal. [Emphasis mine.]
While Tim Bray may be unconvinced, I am a true believer.
I use closures so much that I feel cheated into doing busy work by languages that do not support them. I use continuations less often but frequently enough to appreciate how much time they save me. Neither is strictly required for professional work, but they are potent tools, and a professional who knows how to use them has an advantage over those who do not.
Closures, in particular, are something every professional ought to master. Besides their more celebrated uses, closures make refactoring practical on a small scale. For example, consider the following Ruby method, which we will assume is one of several similar methods belonging to a class that implements some kind of Internet server:
def process
sn = next_serial()
log.info "process/#{sn}: stage 1"
# ... do some work
log.info "process/#{sn}: stage 2"
# ... do some more work
log.info "process/#{sn}: finished"
endThe method first gets a unique serial number, which is used during the processing of requests and also to relate log entries generated by the same processing call. Then the method does its work, logging each stage in passing.
The method makes three logging calls that each hardcode the logger, the logging level, and the format of the log entries. Since these things are repetitive and could very well change, we probably ought to factor them out into an isolated method. After all, we don’t want to rewrite a bucket of logging calls if the log-entry format changes.
Let’s introduce a helper method mylog to hold the common pieces:
def process
sn = next_serial()
mylog("process", sn, "stage 1")
# ... do some work
mylog("process", sn, "stage 2")
# ... do some more work
mylog("process", sn, "finished")
end
def mylog(activity, sn, msg)
log.info "#{activity}/#{sn} #{msg}"
endWhile we managed to isolate the logger, the logging level, and the format of our logging messages, just calling our helper method mylog still requires much redundancy. Worse, the redundancy is on such a low level that we can’t factor it out with another helper method – calling the new helper would be as expensive and redundant as calling mylog directly.
What we need are refactoring tools that scale down to this sub-method level, and that’s where closures come to the rescue. Using them, we can corral the remaining redundancy with a local logging helper llog that “closes over” the relevant state:
def process
sn = next_serial()
llog = lambda { |s| mylog("process", sn, s) }
llog["stage 1"]
# ... do some work
llog["stage 2"]
# ... do some more work
llog["finished"]
endNotice how much simpler and less redundant the logging code is? Each stage can now be logged just by giving its name to llog. We don’t need to pass in the activity name or serial number because llog already knows them both. It knows the activity name because we made it part of llog’s definition, but it knows the serial number because sn is captured in llog’s closure – for free. (Note: If f is a Proc object, f[args] is syntactic sugar for f.call(args).)
Of course, if we have several methods that require a unique serial number and a corresponding logger, we could (thanks to closures) factor things further:
def process
sn, llog = next_serial_and_logger("process")
llog["stage 1"]
# ... do some work
llog["stage 2"]
# ... do some more work
llog["finished"]
end
def next_serial_and_logger(activity)
sn = next_serial()
[sn, lambda { |s| mylog(activity, sn, s) }]
endYou get the point: Closures reduce the cost of working with local state because they capture it implicitly. There is no need to pass the state back and forth; it’s simply there.
Because the craft of programming is dominated by writing the stuff inside of methods, where local state lives, the potential benefits of closures are immense. If more programmers knew how to use them, the profession would be richer for it.
Update 2006-01-11: First, because this article is getting renewed interest, I have turned the comments back on. Second, I edited the article to improve clarity.
Update 2007-05-04: Added syntax highlighting to code snippets.
readers


If someone’s ever iffy on closures, show them an example of a GUI with callback function on events like button press. Using anonymous functions with closures on common data variables, keeping everything in one place without making a one-off class, can be a thing of beauty.
Tom, thanks for the great example. Unlike many of the more academic discussions I’ve seen of closures, using logging as an example gives me something immediately useful to my day-to-day work. Now I’ve got some real motivation to better master closures.
You can also view closures as a safer way of using global data. CS courses have taught us that using global data is bad, kind of like cheating. In your example, the closure routine uses data that actually lives in a ‘frame’ above – may as well be global.
Global data is ‘bad’ because it is error prone. But as any student will tell you, it’s quite good as a way to express logic (otherwise they would not be so tempted to use it).
I don’t think it would require too great a cortortion to use an object as the logger in this situation:
This could have some advantages too: if you want to log to say a file as well as stderr, say, you only need to change the Logger code.
I guess I think of closures are mostly syntactic sugar: they can be mighty sweet, but you don’t get anything you can’t get with objects, particularly if you can define classes at runtime, as I think you can with ruby? (Closures are one-shot objects with only one method.)
To answer Richard Cook, This is from the minimal wxLua example:
Michael S., what closures give you that one-shot objects don’t is implicit capturing of the context. For example, when f is defined below, it closes over x. The x inside of f is the same as the local x:
On the other hand, if I had created an object and initialized it with x, the object would have had a copy of x’s current value, not an implicit connection to the local x variable itself. If x had changed, the object would not have stayed in sync:
In the example I give in the article, this difference isn’t significant because sn doesn’t change over the course of processing a request, but for many kinds of programming this difference means closures will be more convenient than other options.
Just the article I needed! Could you write something similar on continuations? (I get that you can do exception handling with them, but that’s it.)
Tom: doesn’t your example apply if (and only if) x is an integer? If it’s anything else at all it will be passed by reference to the constructor, and therefore modifiable outside the object/closure scope.
i.e. if there was an Integer class, wouldn’t this work?
Michael S., the point of closures is that they capture their environment – the local variables, the variables visible up the scope chain, and in the case of Ruby, the current value of
selfand possibly a locally captured block – so that when the code associated with the closure executes, it does so in the environment in which it was defined.As a programmer using closures, you don’t need to keep asking yourself, What information do I need to pass from the current environment into this block of code so that it can execute properly when the environment has gone away? All you need to do is define the code in the current environment, and the code’s closure will capture the environment automatically. When the code runs, its environment will magically be the environment in which it was defined – not a static snapshot but the real, live thing – and your code will have everything it needs, at no programming cost to you.
In answer to your question, yes, you can take the relevant bits of the current environment and save references to them in an object or some other container and then refer to those saved bits later. (This is, in fact, what closures do behind the scenes.) But it takes effort to identify which bits are relevant, to save the bits, and possibly to refer to the saved bits. Closures eliminate this effort.
Actually it always works, regardless of what x holds. Your example, on the other hand, only works if (1) x holds a value whose type is always referred to by reference and (2) your code never changes x or o.x directly but only talks to the objects contained in these variables by sending messages.
To continue your hypothetical example, if the following line:
were changed to
the connection between o’s x and the current environment’s x would be severed.
Trying to capture environmental bits yourself takes effort and is subject to implementation error. When closures are free and get it right every time, why do it any other way?
Cheers.
—Tom
I think that this “environment capturing” actually breaks encapsulation (closure’s writer has to know that there will be exactly that context) and improves coupling (closure’s user has to provide a certain type of context).
It’s a bit like using “global variables”.
Maybe they can work well for local elaborations, but I can’t see how can they be applied to, for example, libraries construction.
What do you think about this?
@beppegg: I think you’re missing something important: a closure captures the environment in which it is defined, including scoping and visibility. When you call a function that has a closure, then, it will execute in the captured environment and won’t break any encapsulation unless you (the programmer) go out of your way to break it.
In fact, one of the more celebrated uses of closures is to create encapsulation-preserving callback functions. These callbacks wrap around local, protected state in context so that it can be passed safely into the untrusted outside world within an opaque function that cannot be inspected or manipulated to reveal encapsulated details.
Closures are great for writing libraries. Take a look, for example, at the GUI libraries written for languages that support closures; most of the event callbacks rely on closures.
Wow! This is the best example of closures I’ve ever seen, thank you so much for making it clear on why these are so helpful:)
Thanks again.