Dispel the magic by defining the semantics
Posted by Tom Moertel Thu, 23 Jul 2009 19:50:00 GMT
Recently there has been a lot of talk about “magic” in software. Most of the people doing the talking, however, seem to be talking past one another, primarily because they do not share a common understanding of what “magic” means. The few definitions of the term I have seen seem to miss something essential, so I will provide the definition that I think makes the most sense:
In a software system, magic is any behavior whose semantics are poorly defined and unduly expensive to figure out.
Some notes:
- The “unduly expensive” part means that people with different background knowledge are going to find different things magical. If you use certain techniques every day, you will probably find behaviors that make use of those techniques easier to figure out and thus less magical than other programmers might.
- One way, then, to make things less magical is to exhort programmers to get more knowledge. A more practical approach, however, appears in the next point.
- Most “magic” can be de-magicked simply by defining its semantics.
So when I complain about magic in a software system, I am complaining about code whose behavior is not readily discernible. When I did a lot of Rails programming, for example, I always gnashed my teeth when I encountered a function in the Rails API that took a string but did not make clear what that string was supposed to represent. Was it text? Was it HTML? There was only one way to find out: trace through layers of Rails code to figure out what the framework actually did with the string. That is unduly expensive, so I considered such functions to be magic – annoying, waste-my-time magic.
I do a lot more Python programming these days, and while I encounter less magic, I am mystified by the part of Python culture that turns its back on code that is perceived as being overly clever or, to use the popular phrase, not explicit. What is wrong with cleverness, as long as the result is clearly understood? Cleverness, by itself, is not magic. For real magic, you need fuzzy semantics, too.
That is why, when I hear that code is “magical” or “too clever,” I think “poorly defined.” If you define and document the semantics of such code, you dispel its magic. And if you cannot define the semantics, then the code is magical and too clever, and probably ought to be rewritten.
Thus the test for whether something is magical is the clarity of its semantics. Code having clear semantics is not magical. Code having unfathomable semantics is.
readers
Interesting observation but you never really define what you mean by semantics. When I hear semantics in software I think of use cases but this can only highlight aspects and not the entire semantics of a function or operation.
All that you say is true, but I still have this suspicion that the number one cause of magic in software is a misguided attempt to encode DWIM semantics in a situation where WIM is formally inconsistent.
In the popular phrase “not explicit”, what aspect(s) of the code is or is not explicit? The stateful operational steps (from which Backus wished us liberated)? The denotation in an elegant, non-operational math sense? Something else?
Conal, the use of “not explict” is referring to a celebrated principle of Python’s design, enshrouded in The Zen of Python and now firmly lodged in Python programming culture, which states
This principle is what is alluded to when ideas are rejected for being “not explicit.” (A Google search of python.org for PEP and not explicit offers many examples of this usage.)
My take is that the “not explicit” label is applied a bit too freely and sometimes gets in the way of giving full consideration to the merits of ideas. It’s as if the prior knowledge that “not explicit is bad” is strong enough to overshadow a lot of conditional evidence.
In any case, to answer your question, in Python usage, “not explicit” refers to code that has behavior beyond what is revealed by local inspection. Any proposal for control-flow abstractions, for example, is very likely to be labeled “not explicit.”
Operational “behavior” or denotational “behavior”? Operational descriptions are “not explicit” about the denotations they implement. Denotational descriptions are “not explicit” about the operations that implement them.
I wonder if much of this language about explicitness is missing much of the truth, by being not explicit itself, missing the information of (a) explicit about what and (b) explicit to whom? I often see that sort of sloppiness leading to misguided arguments (since people are talking about different things but using language that tricks them into thinking they’re disagreeing over the same thing) and to oppressive social situations (in which focused personal preferences are inflated to general principles/standards).
And thanks for the references.
david karapetyan, what I mean by “semantics” is a clear, complete, and concise description of the behavior of a piece of code. That description can be given in a formal language or just plain words.
As an example, the documentation for Haskell’s Text.ParserCombinators.ReadP library includes a short Properties section that clearly describes how the code behaves. (This description also happens to be executable and provides strong evidence that the code actually does what the description says it does, but this additional feature isn’t necessary, only useful.)
And, just because I know somebody is going to bring it up, I think the typical suite of xUnit-style tests is a poor way to document a system’s semantics. Such tests bury the semantics, being implicit in the tests’ implementation rather than spelled out directly. If you want tests to serve as documentation of semantics, use a tool designed for the purpose, such as QuickCheck – or clear English prose.
Pseudonym, I agree and will go one further: I suspect that the number one cause of magic in software is a misguided attempt to encode DWIM semantics in a situation where WIM is never clear at all, not even in the head of the software’s author, mainly because the author never tried to define the semantics, but instead wrote code and tests directly.
Conal, here behavior refers to operational behavior, based on some informal model of Python execution that most Python programmers hold in their heads.
Hi Tom,
I regret to say that I am the author of the original post that started this whole “magic” good/bad flame war. I was just browsing proggit and came across your post, and I have to complement you on your definition of what magic is. What you’ve said in your post is what I was, unsuccessfully, trying to get across in mine, but just couldn’t find a good way to put it into words. Most of the other comments/posts that I have come across assume that I am referring to Ruby’s ability for metaprograming, malleable syntax, or something along those lines. Truth is, I originally came to Ruby from Lisp, so to be brutally honest, these concepts seem nearly mundane in Ruby. My big pet peeve has never been with the “clever” programming found in many Ruby programs, but instead in the nonintuitive way in which the framework is written. Perhaps it is a mindset that I have, that DHH does not. My background is in classical CS, perhaps his is in design, so it could always be a left vs. right brain thing, but whatever it is, I just don’t seem to grok the thoughts behind why something was done in Rails as easily as I do in other frameworks (Django being my current flavor of the month).
So, thanks for this post, it says exactly what I’ve been trying to say, but with way more success than I could possibly muster on my own.