Tom Moertel's Weblog: Category web development

A bright future: security and modern type systems

Tom Moertel — Wed, 15 Aug 2007 16:07:00 -0400

The recent defacement of the United Nations web site is a prime example of why we programmers shouldn’t trust ourselves to write secure code – at least not without our computers’ help. The U.N. web site, according to Slashdot’s coverage of the incident, was defaced by way of a common, well-known attack: SQL injection. What’s interesting is that programmers can render this attack harmless by employing simple, readily available programming tools such as placeholders and prepared statements. Why, then, are so many web sites, including the UN site apparently, still vulnerable?

Some say it’s because the programmers of these sites are incompetent, but that argument ignores that programmers are human, while the security tools we give them offer meaningful protection only if wielded with inhuman perfection. Having the tools to plug security holes, even if the tools are simple to use and readily available, is not enough to ensure that every single security hole will be identified, let alone plugged. Even the most experienced programmer can be expected to overlook a hole now and then. Unfortunately, one hole is all it takes.

That’s because security is not like other software-quality challenges: its costs are fundamentally asymmetric. For the attacker, the bad guy, the challenge is to find just a single exploitable hole. For us, the good guys, the challenge is to achieve perfection: to plug all of the holes in our code, every single one. That’s because attackers, unlike regular users, can be expected to probe our code until they find a hole to exploit.

How then do we ensure that we have plugged every single hole in our code? Testing isn’t sufficient: we can easily overlook holes when writing tests – a perfectly human error. We could supplement testing with code reviews, painstakingly searching for remaining holes while enforcing the use of hole-preventing best practices, but reviews are expensive and, again, subject to human error. A better approach, both less costly and more reliable, is to delegate this burden to our computers, which can do the job correctly, every single time.

This kind of delegation is possible today with modern static type systems. For example, in A Type-Based Solution to the Strings Problem, I offered a tiny “safe strings” library for the Haskell programming language. The library takes advantage of Haskell’s powerful type system to detect unsafe string interactions at compile time. If we faithfully build our code on top of the library, and our code compiles without error, we can be assured that our code is free – completely free – of SQL-injection (and XSS) holes.

While this result is indeed quite beautiful, it certainly isn’t novel. Researchers have been proving interesting properties via type systems for a long time. As Oleg Kiselyov and Chung-chieh Shan pointed out in a comment on my earlier article, the foundational idea is over three decades old.

More recently, Kiselyov and Shan have extended the idea to guarantee more-interesting properties using a trusted kernel and types that represent lightweight static capabilities. The kernel, which is small enough to be reasoned about and formally verified, carefully hands out capabilities to untrusted application code. The untrusted code, in turn, presents the capabilities back to the kernel to invoke operations, which, thanks to the kernel’s trustworthiness, are guaranteed to be safe. (My safe-string library can be seen as a trivial implementation of this programming style.)

When static type systems are used in this way, they don’t merely catch typos and bugs that good testing would have caught as a matter of course, but offer programmers guarantees that would have been impractical to obtain any other way.¹ If you consider security important, you might bear this fact in mind when choosing languages for your next project.

Going further, the security benefits of rich static type systems are only now starting to trickle into mainstream industry. As libraries like “safe strings” and idioms like static capabilities become more familiar and get woven into future generations of development frameworks, we can expect marked improvements in the security and robustness of our applications.

In the not-too-distant future, perhaps, we might look back in amazement at the days when important security properties were neither free nor guaranteed but expensive and uncertain, underwritten only by the heroic efforts of individual programmers, struggling against impossible odds to achieve inhuman perfection.

Then again, it sure took garbage collection a long time to catch on.

1. How, for example, could you eliminate the possibility of SQL-injection and XSS holes via testing?

I suppose you could do it if you worked at it hard enough. You could augment your string data structures with run-time information about what they represent: this string represents SQL, this string represents plain-old text, and so on. Then you could redefine your string operations and template interpolation systems to assert that their string inputs were compatible. Of course, if these assertions ever failed, they would do so only at run time, when it would be too late to do anything but die rather ungracefully. So you would be forced to augment your code-coverage tools to ensure that every string-path was covered during testing. That way you could catch all potential run-time string failures – indicating holes – during testing and eliminate the holes (and the subsequent need to fail at run time) before you deployed your application for real.

So, yes, you could do it. But to do so would require you, in effect, to write a crude, single-purpose type system that checks types at test time. That says something, doesn’t it?

Don't let password recovery keep you from protecting your users

Tom Moertel — Fri, 09 Feb 2007 15:36:00 -0500

In 2006’s most-read article on my blog, Never store passwords in a database!, I urged web programmers, unsurprisingly, not to store passwords in their user databases. I tried to persuade them to salt and hash the passwords instead: store the salts and hashes in the database and throw the passwords away. The article, posted shortly after the Reddit blog announced the theft of its unprotected user database, generated buckets of comments. Reading over them today, I noticed something that I had missed earlier.

It seems that a decent slice of programmers think that switching to a salted-and-hashed password scheme implies giving up the ability to assist users who have forgotten their passwords. If the passwords are irretrievably hashed away, the programmers reason, there’s no way to recover forgotten passwords and email them to stranded users. Hence those users are screwed.

And that wrinkle, it might seem, is a good reason not to switch to a salted-and-hashed password scheme.

But that wrinkle turns out to be imaginary. Not being able to recover an account’s password does not mean that you can’t recover the account itself. The password, after all, is not the thing of value; the account is. And, as we shall see, we can recover an account without knowing its password.

Recall that the primary benefit of using a hash is that it is a one-way operation. Once you salt and hash a password, there is no practical way to retrieve it. That’s what protects it from would-be attackers. But that also means you can’t get at it, either. Thus sending password reminders to people who have forgotten their passwords is no longer an option.

How, then, can you help your stranded users? One method is to send them account-recovery tokens, which you can think of as one-time, special-purpose passwords. (This method is suitable only if you require no stronger authentication than knowing that your site’s users own the email addresses they claim to own. This is the case for most “low security” sites such as Slashdot, Reddit, and Digg, as well as most blogging systems.)

Here’s how it works. Say Joe has lost his password and can’t log in to your site. He clicks that button that says “I’ve lost my password. Help me!” Now what?

Here’s what you do:

Generate a big, random, unique token and stuff it into Joe’s account record in the database. Stuff the current date and time in there, too.
Send an email to Joe, but instead of enclosing his password (which you can’t recover), tell Joe to click on the enclosed account-recovery link, which includes the random token: http://example.com/recover-account?token=pCIqq1unxntVqc8XtCXg.
Joe receives the email and clicks on the link, which sends his token to your site.
Look up the token in the user database. Is it there?
1. No? Render a screen that says, “Sorry, bub, that token is no longer valid.” Stop.
2. Yes? Excellent. Grab the user record associated with the token. (It will, of course, be Joe’s record.)
Is the date and time stamp on that record more than a few hours old?
1. Yes? Render that screen that says, “Sorry, bub, that token is no longer valid.” Stop.
2. No? Congratulations. Joe has effectively authenticated himself via his email address.
Render a confirmation screen that explains the following to Joe:
1. His account password is going to be reset to the following random string: ocZodbew. (Generate a new random string each time.)
2. If he likes the password, great. If not, he can use the change-password feature immediately after the password is reset.
3. If he understands the above and wants to continue, he should confirm by clicking the big “Reset My Account Password” button.
Joe clicks the button.
You, in response, do the following:
1. Delete the recovery token from Joe’s user record in the database. (This prevents somebody from using the old token to steal his account, should, for example, Joe’s email get stolen.)
2. Replace Joe’s old password with the new, randomly generated password from above. (You will, of course, use the salted-and-hashed method and not store the new password itself.)
3. Log Joe in.
4. Render a screen saying, “Joe, please don’t forget that your new password is ocZodbew. If you would like to change it, just visit Change My Password in your account preferences [provide a link]. Otherwise, you’re logged in and ready to go. Enjoy the site!”
And you’re done.

The code required to make it happen is shorter than the explanation above. It’s one of those easier-done-than-said things.

So, if concerns about account recovery have been holding you back from protecting your users’ passwords, you need hold back no longer. It’s time to “do” your due diligence.

Update 2007-09-10: I made clear that the account-recovery method I describe above is suitable only for low-security sites where a valid email address is sufficient to authenticate users.

Never store passwords in a database!

Tom Moertel — Fri, 15 Dec 2006 13:25:00 -0500

Recently, the folks behind Reddit.com confessed that a backup copy of their database had been stolen. Later, spez, one of the Reddit developers, confirmed that the database contained password information for Reddit’s users, and that the information was stored as plain, unprotected text. In other words, once the thief had the database, he had everyone’s passwords as well.

Had the folks at Reddit salted and hashed the passwords, the thief would now be in a very different situation. Instead of holding all the keys to the kingdom, he would face the prospect of a potentially expensive search for each and every user’s password he wanted to extract from the database. The expense of the search would likely have dissuaded him from making the attempt in earnest, given how little exploitable value a Reddit account represents. In short, the passwords would have been secure, even though the database had fallen into the thief’s hands.

Why, then, didn’t Reddit’s programmers salt and hash the passwords before storing them in their database? Because, according to the earlier post by spez, they wanted to be able to send forgotten passwords to users via email. It was a design decision: they weighed the risks of having plain-as-day passwords in the database against the convenience of being able to email users their forgotten passwords and decided that, in the balance, convenience carried more weight. It’s a decision they now regret. (It’s a doubly unfortunate decision because you don’t need to store passwords in your user database in order to offer convenient account recovery.)

The reason I’m writing about this event isn’t to kick the good folks at Reddit while they’re down. Rather, I’m trying to make a point:

If you are storing passwords in a database, you are almost certainly making a mistake.

The guys at Reddit are known for being smart. They thought they had a good reason for storing passwords in their database. They were wrong. If smart programmers can make this mistake, lots of programmers can. Do you think you have a good reason for storing passwords in your database? If so, you’re probably wrong, too.

How can I be so sure? Because, when it comes to web-app authentication, cutting corners doesn’t buy you anything. It doesn’t save you coding time. It doesn’t give your users a better experience. All it does is weaken the security of your web application, needlessly putting your users, your employer, and yourself at risk.

So please let me take this opportunity to ask if you know of (or perhaps work on) any software systems that store passwords as plain, unprotected text in a database. If so, fix your software now:

Salt and hash each and every password (use an expensive hashing function such as bcrypt that was designed for password applications)
Store the salt and hash – not the password – in your database.
Throw the password itself away.

You’ll be glad you did.

Update: Minor edits for clarity.

Update 2007-02-13: Salting and hashing does not get in the way of account recovery. You do not need to email users their forgotten passwords: there are other account-recovery options that are just as convenient but much more secure. See Don’t let password recovery keep you from protecting your users for more.

Update 2007-10-03: Revised text slightly to emphasize that there is no benefit to be had by implementing a weak password system, and therefore there is no reason not to implement a secure system. Pointed more directly to bcrypt, too.

A type-based solution to the "strings problem": a fitting end to XSS and SQL-injection holes?

Tom Moertel — Wed, 18 Oct 2006 21:40:00 -0400

Even skilled programmers have a hard time keeping their web applications free of XSS and SQL-injection vulnerabilities. And it shows: a sobering portion of web sites are open to some scary security threats.

Why are so many sites vulnerable to these well-known holes? Probably because it’s insanely hard for programmers to solve the fundamental “strings problem” at the heart of these vulnerabilities. The problem itself is easy to understand, but we humans aren’t equipped to carry out the solution. Simply put, we just plain suck at keeping a bazillion different strings straight in our heads, let alone consistently and reliably rendering their interactions safe whenever they cross paths in a modern web application. It’s easy to say, “just escape the little buggers,” but it’s hard to get it right, every single time.

Computers, on the other hand, are pretty good at keeping track of details by the bucket-full. Wouldn’t it be nice, then, if our programming languages gave us the power to delegate this nasty “strings problem” to our computers, which could then devote their unwavering mechanical precision to grinding the problem out of existence? Isn’t that the kind of thing modern programming languages are supposed to be good at?

I’d like to think the answer to that question is a big, you betcha.

So let’s grab a modern programming language and solve the strings problem.

Let’s solve the strings problem in Haskell

In this article, we will look at one way (among many) to solve the strings problem: by adding Ruby-style string templates to Haskell. These templates support “interpolation” via the usual, convenient #{var} syntax, but here interpolation is type safe. Haskell’s type system will prevent us from inadvertently mixing incompatible string types, and it will detect mistakes at compile time, before they can become live XSS or SQL-injection holes. Further, our solution will offer us these benefits without making us jump through hoops or pay some onerous syntax penalty.

To be more specific, the system offers the following benefits:

It provides a string-management kernel that lets you create “safe strings” by certifying a regular string as representing either text or a fragment of a known language.
It allows you to conveniently define new language types for any string-based language that you can provide an escaping rule for (e.g., XML, URLs, SQL, untrusted user input).
It provides compile-time syntactic sugar (via Template Haskell) that makes working with safe strings as convenient as working with string interpolation in languages like Ruby and Perl.
It catches and reports (at compile time) the following commonly made programming errors:
- failing to escape a plain-old-text string before mixing it into a string that represents a language fragment
- mixing strings that represent fragments of incompatible languages
- mixing strings that represent fragments of compatible languages in an ambiguous way (the system will force you to disambiguate)

(This is a long one, so grab an espresso, lean back, and read on in style. Also, if you have a smoking jacket, you might want to get it now.)

Before I describe this Haskell-based solution, let’s take a closer look at the strings problem and review why a type-based approach makes sense. (If you already understand the strings problem and are convinced that it is both important and tricky to solve, feel free to skim the first third of this article.)

Examining the “strings problem”

Most web applications are just business-logic-driven string processors. They take strings from user-submitted forms, database queries, web-service responses, templates, and myriad other sources, and they combine the strings to generate yet more strings, which they emit as output and fling across the Internet, into your web browser.

For example, consider this snippet of Ruby (on Rails) code that I used to add submit-to-Reddit and submit-to-del.icio.us buttons to articles on my blog:

def submit_this_article_links(article)
  site_list(article).map do |submit_title, submit_url, image_tag|
    %(<a href="#{h submit_url}" 
         title="#{h submit_title}: &#x201C;#{h article.title}&#x201D;" 
      >#{image_tag}</a>)
  end.join("&#160;")
end

def site_list(article)
  u_title = u(article.title)
  u_url = u(url_of(article, false))
  [  # I really belong in a database table
    [ "Submit to Reddit.com",
      "http://reddit.com/submit?url=#{u_url}&title=#{u_title}",
      image_tag("reddit.gif", :size => "18x18", :border => 0)
    ],
    [ "Save to del.icio.us",
      "http://del.icio.us/post?v=2&url=#{u_url}&title=#{u_title}",
      image_tag("delicious.gif", :size => "16x16", :border => 0)
    ]
  ]
end

When writing this code, I had to keep track of at least three different kinds of strings:

Plain-old text, e.g., article titles
URLs, e.g., article permalinks
XHTML fragments, e.g., the hypertext link to Reddit’s submission form

In code like this, each type of string must conform to the requirements of its own little language, and it’s the programmer’s job – your job – to make sure that differences in these requirements are accounted for when combining strings. Getting it right is a difficult trick to pull off, and getting it right consistently is something even the best developers have difficulty doing.

In the tiny snippet of code above, for example, I had to remember to do all of these things:

URL-escape (using the u helper method) the article’s title before inserting it into the submit-URL template
URL-escape the URL for the article’s permalink before inserting it into the submit-URL template
HTML-escape (using the h helper method) the final, expanded submit-URL template before inserting it into the hypertext-link template
HTML-escape the submit-title (e.g., “Submit to Reddit”) before inserting it into the hypertext-link template
HTML-escape the article’s title before inserting it into the hypertext-link template

That’s a lot to keep track of when coding.

But that’s not all. I also had to know not to escape the result of calling image_tag, because that helper method returns an HTML fragment, which is already in the language of the hypertext-link template into which it is inserted. Escaping it would have turned the image-element markup into embedded text that happens to look a lot like HTML markup.

And that’s not the worst of it. If you screw up any one of these steps for the typical web application, you open the door to a host of nasty problems. If you’re lucky, the damage will be contained to broken links or a rendering problem that most people won’t notice, maybe a weird database error now and again. In the worst case, however, you’re screwed: Your application’s customers become vulnerable to cross-site-scripting (XSS) attacks and your database is opened to injected SQL, through which enterprising crackers might steal your customers’ account data or do even nastier things.

Clearly, the strings problem is common enough and nasty enough to merit our attention. Many of our favorite problem-stomping practices, however, have not proved effective on the ever-tricky strings problem.

Unit testing is an inefficient solution to the strings problem

Unit testing is one of the most efficient programming practices for increasing the quality of software. If you write unit tests pervasively as you code, you are likely to nip many kinds of programming problems in the bud, saving time and effort, which you can then re-invest in your code. Further, unit-testing suites make for swell regression-detection nets and thus free you to refactor crufty code without fear of introducing breakage elsewhere. As a result, you’re more likely to keep your code lean and mean.

Despite its general effectiveness, unit testing is an inefficient way to defend against the perils of the strings problem. That’s because the strings problem is caused by knowledge deficits, which you can’t test for. If you don’t realize that you must escape one URL before you stuff it into another URL, you probably won’t think to write tests for that requirement.

Moreover, if you do think to write the tests, it’s expensive to get them right. In most unit testing scenarios, getting the tests right is usually easier or at least comparable in difficulty to getting the code that’s being tested right. That’s why unit testing is usually so efficient. For the strings problem, however, getting the tests right is often much more expensive than writing typical string-handling code. In my code sample above, for example, there are at least six ways the strings problem can cause trouble. How do you test for them all without making a mistake? It’s not easy.

In sum, unit testing probably isn’t the answer to the strings problem.

Eating and having one’s cake: a type-based solution

An ideal solution would combine the safety of the question-marks solution with the straightforward convenience of string interpolation, and it would work for all kinds of strings, not just SQL, and, because I’m implementing it in Haskell, it would lovingly nestle into Haskell’s type system and gain the full benefits of type-inferencing goodness.

How would it work? Well, let’s back up and think about strings for a moment. We can divide strings into two classes: (1) those that represent text, in which every character represents literally itself; and (2) those that represent fragments of interpreted languages, such as XML or SQL, where each character’s interpretation depends on the rules of the associated language. In text, for example, an ampersand (“&”) represents an ampersand, but in XML an ampersand represents the start of a character-entity reference.

It doesn’t make sense, then, to join text strings directly with language-fragment strings. If you did join them, text characters could be misinterpreted as language characters. For the same reason, it doesn’t make sense to join fragments of different languages together. (It does make sense, however, to escape text strings or language fragments “into” a target language and then join them with strings in the target language.)

A sound solution, therefore, should enforce the following fundamental, safe-string-handling rule: Do not allow strings that represent fragments of one language to be directly joined with strings that represent either plain text or fragments of another language.

The trick is making the computer enforce this rule for us. As it turns out, modern type systems absolutely love to do this kind of thing.

A solution to the strings problem in Haskell

Making the computer enforce our safe-string-handling rule in Haskell is fairly easy. All it takes is a little code. (As we go through the following code, remember that we’re writing a library. Normally, as users of the library, this code would be invisible to us.)

To begin, we create a module for our code and export the essential types and functions that make up our about-to-be-written safe-string kernel:

module SafeStrings
(
  Language(..),
, SafeString -- we export the data type but not the constructors
, empty, frag, text
, cat, (+++)
, render, renders, lang
, q
, declareSafeString
)
where

In order to create safe strings that correspond to particular languages, we need to tell the computer what we mean by Language:

class Language l where
    litfrag  :: String -> l   -- String is a literal language fragment
    littext  :: String -> l   -- String is literal text
    natrep   :: l -> String   -- Gets the native-language representation
    language :: l -> String   -- Gets the name of the language

Here we’re saying that Language is the class of languages, i.e., all data types l for which we can provide four functions:

litfrag – converts a string that represents a language fragment into a language fragment
littext – converts a string that represents plain text into a language fragment that represents the text (via escaping)
natrep – converts a language fragment, verbatim, into a string that represents the language fragment
language – returns the name of the language associated with a given fragment

Further, we need to declare a few “language laws” that conforming Language types must obey. These laws are for us. They will keep us honest when teaching the computer about new languages. Here are the two laws we will require language types to satisfy:

natrep (litfrag s) == s
natrep (littext s) == (escape_L s)

The first law requires that (natrep . litfrag) be equivalent to the identity function for strings. The second law requires that (natrep . littext) be equivalent to the text-escaping function for a given language L. For example, for the language XML:

natrep (litfrag "<em>wow!</em>") ==> "<em>wow!</em>" 
natrep (littext "ham & eggs")    ==> "ham &amp; eggs"

Next, let’s construct a type-safe container for strings having a known language:

data Language l => SafeString l
    = SSEmpty
    | SSFragment l
    | SSCat (SafeString l) (SafeString l)

This data-type definition says that if l is a language, we can construct SafeString values for that language. Each value can represent an empty fragment of the language (via SSEmpty), a non-empty fragment of the language (via SSFragment), or the concatenation of two other SafeString values for the language (via SSCat).

Now comes the interesting part. We are going to leverage the type system to enforce the safe-string-handling rule for us.

We will do this using the SafeString data type we just defined. We have already placed the data type’s definition into a module that does not export the type’s data constructors. That means we will not be able to create SafeString values for ourselves. Instead, we must ask a small set of kernel functions, which are exported, to create the values on our behalf.

These kernel functions, which we are about to write, will create SafeString values only in accordance with our safe-string-handling rule. In particular, they will require us to certify that an existing string represents either text or a language fragment before creating a corresponding SafeString value for us. From then on, the type system will know which language the string is associated with and prevent us from joining it to regular strings or to SafeString values associated with other languages.

Let’s write these constructor functions now:

empty      :: Language l => SafeString l
empty       = SSEmpty

frag, text :: Language l => String -> SafeString l
frag f      = SSFragment (litfrag f)
text s      = SSFragment (littext s)

Here’s what the functions do:

empty – creates an empty SafeString in the Language l
frag f – takes a string that you certify as representing a fragment in the Language l and returns a corresponding SafeString
text s – takes a string that you certify as representing text and returns a corresponding SafeString in the Language l

Once the kernel creates SafeString values for us, we need some way to combine them safely. Thus we define the (+++) operator and the cat function:

-- join two SafeStrings of the same language
(+++) :: Language l => SafeString l -> SafeString l -> SafeString l
(+++)  = SSCat

-- join a list of same-language SafeStrings
cat   :: Language l => [SafeString l] -> SafeString l
cat    = foldr (+++) empty

Finally, we need a way to convert SafeString values into normal strings so that we can pass them through the boundaries of our safe-string-protected code and into the outside world. For this, we write the render function:

render ss = renders ss ""

renders SSEmpty        = id
renders (SSFragment a) = (natrep a ++)
renders (SSCat l r)    = renders l . renders r

(Don’t worry about the renders stuff. It implements a Haskell idiom for fast string concatenation.)

As a convenience, let’s round out our kernel with a Show instance that tells Haskell how to format SafeString values for display.

instance Language l => Show (SafeString l) where
    showsPrec _ ss =
        (lang ss ++) . (":\"" ++) . renders ss . ('"':)

lang ss =
    let SSFragment e = ss in language (undefined `asTypeOf` e)

And that’s our SafeStrings kernel.

Another look at the SafeStrings kernel

The following illustration, complete with poorly chosen colors, provides a visual summary of our system:

(Don’t worry about the $(q ...) stuff for the moment, we’ll talk about it later.)

Activating our mad art-interpretation skillz, we can now decipher the illustration:

Regular strings gain “admittance” to the SafeStrings kernel only via the text and frag certification functions, which we use to create corresponding safe strings for a given language. Once created, the safe strings live their entire lives in the fleshy-colored, egg-shaped protective sac that is the kernel, whose safe-string functions and operators use Haskell’s type system to prevent us from accidentally mixing the strings in unsafe ways. Further, because the kernel does not export its underlying data structures, we can’t screw around with the innards of our safe strings to break the kernel’s promises. When our safe strings have finally reached their ultimate, beautiful state, we can render them into regular strings and pass them bravely into the cruel outside world – where, most likely, somebody else’s broken code will screw them up anyway. But at least we tried.

Our first SafeString module: SafeXml

Now that we have written our SafeStrings kernel, let’s use it to create a SafeXml module that we can use for working with XML. Again, we will be writing library code that under normal circumstances would be hidden from view.

First, we will create a new module that uses the SafeStrings kernel:

module SafeXml
( Xml, xml, renderXml, module SafeStrings )
where
import SafeStrings

Next, we will create a wrapper type to testify that a string represents a fragment of XML:

newtype XmlString
    = XmlString { unXmlString :: String }
    deriving Show

If you go back and look at the export list for the module, you’ll see that the XmlString data type is not exported. It is internal to the module, and thus we, as clients of the module, can’t create values of that type. That means we can’t “forge” XML strings into existence. We can create them only through the safe-string kernel, and even then only by certifying a regular string as representing text or a language fragment. (The kernel, in turn, will create the needed values through the Language interface, which we now discuss.)

Like all good language types, XmlString needs to be a member of the Language type class, so we provide the necessary instance functions:

instance Language XmlString where
    litfrag  = XmlString
    littext  = XmlString . escapeXml
    natrep   = unXmlString
    language = const "xml"

Note that the functions satisfy the language laws we defined earlier. (The proof follows immediately from the definitions of XmlString, unXmlString, and escapeXml.)

Next, we need to write a function to implement the escaping rule for XML:

escapeXml xs =
    concatMap esc xs
  where
    esc '<'  = "&lt;"
    esc '&'  = "&amp;"
    esc '"'  = "&#34;"
    esc '\'' = "&#39;"
    esc x    = [x]

Next, because we expect to work with XML frequently, we will create a convenient type synonym, Xml, for SafeString values that represent XML:

type Xml = SafeString XmlString

Finally, we will create a few convenience functions to create and render XML fragments. These functions are identical to the SafeString kernel’s frag and render functions but for the Xml type exclusively. When we use these functions, we won’t need to provide additional type annotations; the computer will know we are dealing with XML strings:

xml :: String -> Xml
xml = frag

renderXml :: Xml -> String
renderXml = render

And we’re done.

Before going on, let me point out two things:

If you think the code we have written so far is long or perhaps confusing, please remember that it is library code. Typically, you would never see it. All you would do is import SafeXml and start using the library.
The SafeXml implementation is formulaic, and we can replace all of it except for the escaping function’s definition with a single line of code, something we will do later.

A quick test drive of our SafeXml module

Let’s give our SafeXml module a spin in the GHC interactive shell.

We can create an XML fragment by certifying that a regular string represents a language fragment (via the frag function) and telling Haskell that we expect a result of type Xml.

Ok, modules loaded: SafeXml, SafeStrings.
*SafeXml> frag "<em>wow!</em>" :: Xml
xml:"<em>wow!</em>"

Note how the output is prefixed with the label “xml:” to tell us that our kernel certifies this value to represent an XML fragment.

Because entering type annotations can be inconvenient, we can instead use the xml function, which certifies a string not just as a fragment but as an XML fragment:

*SafeXml> xml "<em>wow!</em>" 
xml:"<em>wow!</em>"

If we want to represent text in XML, the kernel will automatically escape it for us:

*SafeXml> text "ham & eggs" :: Xml
xml:"ham &amp; eggs"

Now let’s try to do something naughty. Will the type system let us?

*SafeXml> let someXml = xml "<em>Hi!</em>" 
*SafeXml> let plainOldText = "ham & eggs" 
*SafeXml> someXml ++ plainOldText

<interactive>:1:0:
    Couldn't match `[a]' against `Xml'
      Expected type: [a]
      Inferred type: Xml
    In the first argument of `(++)', namely `someXml'
    In the definition of `it': it = someXml ++ plainOldText

In Haskell, the (++) operator is used (among other things) to join strings. In the code above, we tried to use this operator to join an XML fragment to a plain-old string, which would have violated our safe-string-handling rule. Fortunately, we were unable to fool the type system into allowing this ill-conceived union to occur. Note that our mistake was caught at compile time, before the code was ever converted into executable form.

Perhaps we can persuade our newly-defined (+++) operator to make the union:

*SafeXml> someXml +++ plainOldText

<interactive>:1:12:
    Couldn't match `SafeString XmlString' against `[Char]'
      Expected type: SafeString XmlString
      Inferred type: [Char]
    In the second argument of `(+++)', namely `plainOldText'
    In the definition of `it': it = someXml +++ plainOldText

Again, the type system has prevented us from doing something naughty. If, however, we certify that the plain-old string represents text, we can make a safe union:

*SafeXml> someXml +++ text plainOldText
xml:"<em>Hi!</em>ham &amp; eggs"

Syntactic sugar for safe strings

Not having to worry about the strings problem anymore is fabulous and all, but having to type in frag, text, and +++ is kind of clunky. Let’s get rid of the clunkiness by introducing some syntactic sugar.

The common case when dealing with strings in web applications is templates. For example, here’s a simplified version of the link_to method from the deservedly popular Ruby on Rails. The method wraps a hypertext link around some content by “interpolating” the content and a URL into a link template:

# NOTE: this example is in Ruby

def link_to(content_xhtml, url)
  "<a href=\"#{h url}\">#{content_xhtml}</a>" 
end

In this code, we need to HTML-escape the URL (via the h helper) before interpolating it into the template. We do not need to escape the content, however, because it is already in the template’s language, XHTML.

Now, to introduce our syntactic sugar, here’s link_to rewritten in Haskell and using safe strings:

-- Haskell code

link_to :: Xhtml -> Url -> Xhtml
link_to content url =
    $(q "<a href=\"#{r url}\">#{=content}</a>")

The type signature makes clear to everybody that the content parameter is XHTML, the url parameter is a URL, and the result is XHTML. The signature isn’t needed, but link_to is the stuff of libraries, and so annotations are good form.

The interpolation syntax is like Ruby’s, but with slightly different modifiers:

The template-quoting syntax is $(q "this is a template"). (Mnemonic: q for quote).
Within a template, we can interpolate variables using the familiar #{var} syntax.
If an interpolated variable holds a plain string, it will be escaped into the template automatically.
If an interpolated variable holds a safe string, we must use an interpolation modifier to specify how it should be interpolated (to avoid ambiguity):
- #{r var} renders the safe string in var into text, and then interpolates the text into the template, escaping as necessary (mnemonic: r for render).
- #{= var} inserts the safe string in var directly into the template, which must be of the same language (mnemonic: = for equal language types).
As a bonus, #{s var} interpolates any Show-able value in var into the template as text, escaping as necessary.

It’s pretty easy to tell which interpolation option is right for any situation, but late-night coding sessions make fools of us all. That’s why the type system is there to catch us when we make a dumb mistake.

Let’s try out the sugary link_to method:

> link_to (text "Tom's Weblog") (url "http://blog.moertel.com/")
xml:"<a href="http://blog.moertel.com/">Tom's Weblog</a>"

Let’s take advantage of type inferencing in the next example:

> link_to $(q "<em>Espresso!</em>")
          $(q "http://google.com/search?q=espresso&oe=utf-8")

xml:"<a href="http://google.com/search?q=espresso&amp;oe=utf-8">
     <em>Espresso!</em></a>"

In the above example, we supplied templates as input parameters. Haskell figured out their types and took care of the escaping (or not escaping) for us.

Now that we know what the syntactic sugar looks like, let’s see how to implement it.

Implementing the syntactic sugar using Template Haskell

We implement the SafeString library’s syntactic sugar using Template Haskell. A small function q (for “quote”) parses the sugared syntax at compile time and emits equivalent code using our safe-string functions frag, text, and so on. For example, the following sugar:

$(q "<em>#{mystr}</em>")

becomes the following code:

cat [frag "<em>", text mystr, frag "</em>"]

The code that makes it happen is fairly straightforward if you know Template Haskell, so I’ll skip the explanation because this article is already way too long. As usual, it’s library code, so normally we wouldn’t see or care about it. All we care about is the $(q "...") sugar that the code makes available to us.

Here it is:

import Language.Haskell.TH
import qualified Text.ParserCombinators.ReadP as P

-- Convert template sugar into calls to frag, text, cat, etc.
-- This function is exported by the SafeStrings module.

q spec =
    [| cat $(parts) |]
  where
    parts = case xparse spec of
        []   -> error ("bad template: " ++ show spec)
        ps:_ -> foldr gen [| [] |] ps
    gen p ps' = (\p' -> [| $p' : $ps' |]) $ case p of
        SFrag s  -> [| frag $(litE (stringL s))         |]
        SIFrag s -> [| $(varE (mkName s))               |]
        SIShow s -> [| text (show $(varE (mkName s)))   |]
        SITxt s  -> [| text $(varE (mkName s))          |]
        SIRTxt s -> [| text (render $(varE (mkName s))) |]

-- AST for template-specification parts

data SpecPart
    = SFrag String  -- ^ language fragment
    | SIFrag String -- ^ insert fragment by variable reference
    | SIShow String -- ^ insert rendered variable via show
    | SITxt String  -- ^ insert literal text variable
    | SIRTxt String -- ^ insert rendered safe string var as text
  deriving Show

-- Parse a template specification

xparse spec = do
    (result, "") <- P.readP_to_S templateP spec
    return result
 where
    templateP = do
        P.many ((liftM SFrag (P.munch1 (/= '#'))) P.<++
                interpolationP P.<++
                liftM SFrag (P.string "#"))

    interpolationP = do
        P.string "#{"
        spec <- P.manyTill P.get (P.char '}')
        return $ case spec of
          'r':' ':var -> SIRTxt (strip var)
          's':' ':var -> SIShow (strip var)
          '=':var     -> SIFrag (strip var)
          var         -> SITxt  (strip var)

strip = frontAndBack (dropWhile (== ' '))
frontAndBack f = reverse . f . reverse . f

More sugar: defining additional safe-string types

One additional bit of Template Haskell code, which I won’t reprint here, defines declareSafeString. This function lets us eliminate the boilerplate code when defining new safe-string types. For example, compare our earlier definition of the SafeXml module with the following implementation of a module for safe URL strings:

module SafeUrl (Url, url, renderUrl, module SafeStrings) where
import SafeStrings
import Text.Printf
import Data.Char (ord)

escapeUrl xs =
    concatMap esc xs
  where
    esc x | isReserved x || x > '~' = urlEncode x
          | x == ' '                = "+"
          | otherwise               = [x]

urlEncode x  = '%' : printf "%02x" (ord x)
isReserved   = (`elem` "!#$&'()*+,/:;=?@[]")

$(declareSafeString "url" "Url" [| escapeUrl |])

The final line generates the boilerplate code for the wrapper type, the language definition, the Url type synonym, and the url and renderUrl language-specific convenience functions.

One big example to wrap things up

Because we have been discussing mainly library code, let’s take a step back and see some typical user-level code that uses safe strings. After all, that’s what counts.

Here is a Haskellized, safe-strings version of the Ruby (on Rails) code that I presented at the beginning of the article to add submit-to-Reddit and submit-to-del.icio.us buttons to my blog:

module Example where
import List (intersperse, break)
import SafeXml
import SafeUrl

type Xhtml = Xml

submit_this_article_links :: Article -> Xhtml
submit_this_article_links (Article title url) =
    cat . intersperse nbsp $ do
    (submit_title, submit_url :: Url, image_tag) <- site_list
    return $(q
      "<a href=\"#{r submit_url}\" \
         \title=\"#{submit_title}: &#x201C;#{title}&#x201D;\" \
        \>#{=image_tag}</a>" )

  where

    nbsp = xml "&#160;"

    site_list = [  -- move me into a database table
      ( "Submit to Reddit.com"
      , $(q "http://reddit.com/submit?url=#{r url}&title=#{title}")
      , image_tag "reddit.gif" "18x18" 0
      ),
      ( "Save to del.icio.us"
      , $(q "http://del.icio.us/post?v=2&url=#{r url}&title=#{title}")
      , image_tag "delicious.gif" "16x16" 0
      ) ]

The code looks fairly similar to the original Ruby code, with the exception of some extra backslashes, courtesy of Haskell’s rather-unfortunate syntax for multi-line string constants. (Perl and Ruby’s <<HERE syntax would be a welcome addition.)

The other big difference is that, in this version, the type system has automatically checked the code for strings-problem errors.

For completeness, here is the example’s supporting code (again modeled on Ruby on Rails). This code also makes extensive use of safe-string templates:

image_tag :: String -> String -> Int -> Xhtml
image_tag file_name size border =
    $(q "<img src=\"#{r image_url}\" height=\"#{height}\" \
         \width=\"#{width}\" border=\"#{s border}\"/>")
  where
    image_url         = $(q "#{=site_root}images/#{file_name}")
    (width, _:height) = break (=='x') size

link_to :: Xhtml -> Url -> Xhtml
link_to content url =
    $(q "<a href=\"#{r url}\">#{=content}</a>")

data Article = Article
  { article_title  :: String
  , article_url    :: Url
    -- more fields here
  }

sample_article =
    Article "I love chunky bacon!" $
    url "http://blog.moertel.com/permalink/to/article"

site_root :: Url
site_root =  url "http://blog.moertel.com/"

Have we done it? Have we learned anything?

Have we rid ourselves of the strings problem? If we use a programming language like Haskell and a library like SafeStrings, I think we can answer yes.

To be clear, the fundamental problem of having to manage different kinds of strings is still with us. As programmers, we still must understand the differences between URLs, XML, SQL, untrusted user input, and so on. But now, we don’t have to be perfect. As long as we can reliably slap the right type on a string when it first appears, we can let the computer worry about it from then on. If we forget to escape the string later, as it winds its way through the twisty code of a large web application and interacts with other strings in potentially dangerous ways, the computer will catch our mistake – at compile time, before it can possibly become a live security hole.

But if slapping the right types on strings – certifying them – is a pain in the neck, we won’t do it. We will happily go back to our days of winging it, where every string interaction becomes an opportunity for a perfectly human mistake to give birth to a nasty security vulnerability.

That’s why syntax matters. That’s why Template Haskell, Lisp macros, and other meta-programming tools are important: they let us craft friendly syntaxes that encourage the use of programming aids like SafeStrings. That’s why type inferencing is important: it lets us do away with redundant annotations and makes working with types convenient, so we can reap the benefits of strong guarantees without having to pay prohibitive costs.

If there is a moral to this story, it’s that modern type systems and macro systems are powerful tools. They let us do things that otherwise would be impractically inconvenient. They extend our reach as programmers and let us solve problems that we couldn’t solve before. Why, then, do so many programmers dismiss these tools as mere academic curiosities? Why do so many programmers turn away to fight unaided against, and frequently lose to, the very problems that these tools could so easily solve?

Update: minor edits for clarity.

Ryan Carson: Building web applications on a budget

Tom Moertel — Wed, 08 Feb 2006 14:25:00 -0500

Via Simon Willison’s post about the 2006 Future of Web Apps Summit, I found notes for Ryan Carson’s talk about building web apps on a budget. In the talk Ryan breaks down the budget for starting up DropSend:

Budget (£)	Need
5,000	Branding & UI design
8,500	Development of web app (developers also given small equity stake)
2,750	Desktop apps (Windows and Mac)
1,600	Building XHTML/CSS
500	Hardware (internal development server)
800	(per month) hosting and maintenance
2,630	Legal fees
500	Accounting fees
500	Linux-specialist fees
1,950	Misc. fees (trips, replace broken hardware)
250	Trademark
200	Merchant account
500	Payment processor’s setup fee
25,680	Total

That is about $45K in US dollars. In other words, you can launch a new web application for less than a skilled technology worker’s salary. Or, if you are a skilled technology worker, you can do much of the work yourself and launch a new web application for about $25K.

Got an itch to scratch?

A simple Apache recipe for migrating blog articles to a new host

Tom Moertel — Mon, 06 Feb 2006 17:52:00 -0500

In Everything old is new again, I wrote that I was moving articles from my old blog over to my new Typo-powered blog (here). Now that the process is underway, I need to make sure that people looking for my old articles can find them at their new home. To solve this problem, I am using a simple Apache httpd recipe to redirect requests for the old articles to the corresponding updated articles on my new blog. In case you need to do something similar some day, here is the recipe.

First, set up a mapping file

Create a two-column mapping file that you can use to map each article’s old location to its corresponding new location. If there are any parts of the locations that never change, you can factor them out to reduce clutter.

For example, the article “My New Radio VCR” has the following old and new locations (the constant parts are emphasized):

Old	= http://community.moertel.com/ss/space/2004-02-20
New	= http://blog.moertel.com/articles/2004/02/20/my-new-radio-vcr

Its entry in my mapping file looks like this:

# File: /path/to/conf/old-blog-to-new.txt
# Map articles from old blog to new blog.
#
# OLD LOCATION    NEW LOCATION
# .../ss/space/X  http://blog.moertel.com/Y

...               ...
2004-02-20        articles/2004/02/20/my-new-radio-vcr
...               ...

Second, configure Apache to use the mapping file

Edit the Apache configuration that controls the old locations. Add a set of mod_rewrite rules to match requests for the old locations and redirect them to the corresponding new locations, using the mapping file as a reference. For example, here is my configuration:

# in Apache's configuration for community.moertel.com

RewriteEngine on
RewriteMap blogmap txt:/path/to/conf/old-blog-to-new.txt
RewriteCond ${blogmap:$1|NOT-FOUND} !=NOT-FOUND
RewriteRule ^/ss/space/(.+) http://blog.moertel.com/${blogmap:$1} [R=301,L]

The first line makes sure that mod_rewrite is active.

The second line tells Apache to load the mapping file. Apache will cache the mapping file’s contents for speed, but it is smart enough to reload the file when modified. That means you can add new entries to the mapping file at any time, and Apache will act on them immediately, no restart or reload required. Every time I moved an article over to the new blog, for example, I just edited the mapping file, and the new location “went live,” replacing the old.

The third line says that the recipe is conditional upon there being a matching entry in the mapping file. If no entry exists, the recipe will not apply, and the request will be handled as usual.

The final line defines the rewrite rule. In this example, it tries to match requests that start with ”/ss/space/X_”, where _X is any suffix. (The prefix “http://community.moertel.com” is implied because this configuration is for the community.moertel.com site.) If the request matches, X is stored in the $1 variable. Then – and this is one of those things that makes mod_rewrite seem tricky – the condition defined in the previous line is tested using the current value of $1. If the condition is satisfied, the request is redirected to http://blog.moertel.com/Y_, where _Y is the corresponding location for X, according to the mapping file.

The [R=301,L] part of the rewrite rule is important. It specifies that redirects should be of the 301-Permanent variety. This advertises to the world that the new locations are intended to replace the old locations. Using permanent redirects also ensures that any Google juice that may have accumulated for my articles follows them to their new home.

Third, activate the new configuration

This part is easy: restart Apache to make sure it turns on the rewrite engine and activates the new configuration directives.

Finally, test it out

To see if everything is working properly, visit an article’s old location to see if you are redirected to the corresponding new location. For example:

http://community.moertel.com/ss/space/2004-02-20

If you click on this link, you should be redirected to blog.moertel.com.

And that’s the recipe.

Top-ten weblog usability mistakes: My blog's scorecard

Tom Moertel — Wed, 26 Oct 2005 16:51:00 -0400

Jakob Nielsen’s Alertbox for 17 October 2005 is Weblog Usability: The Top Ten Design Mistakes. In other words, it’s a top-ten list of things not to do on your blog if you care about usability.

Since I care about usability, I decided to test my own weblog against Jakob’s top-ten list. How did I fare?

Let’s see:

Top Ten Weblog Usability Mistakes

No author biography: Fail. Oops, got me there. While I have a bio on the Community Projects site, I don’t even link to it from my blog.
No author photo: Fail. Oops, again. I don’t have a photo of myself on the blog, nor on any other site (that I know of).
Nondescript posting titles: Pass. I generally give my posts informative titles such as How to change symlinks atomically and Simple data formats are not going away. Only rarely do I use cutesy titles such as On the effortless cultivation of humility, where readers are forced to guess what the post is about.
Links don’t say where they go: Pass. As a reader, I find “click here” links to be annoying, and so I avoid the practice in my writing.
Classic hits are buried: Pass. I link to popular topics in the Popular Topics sidebar.
The calendar is the only navigation: Pass. My posts are organized by topic as well as by date. Further, the ever-present live search makes finding posts by content easy.
Irregular publishing frequency: Fail. I don’t have a regular posting schedule. When work gets heavy, for example, I rarely post.
Mixing topics: Semi-Pass. I do mix up topics somewhat, but almost all of my topics fall into the category of “stuff programming geeks like” and in that regard are fairly consistent.
Forgetting that you write for your future boss: Pass. I don’t think there is anything on my blog that a future employer would find troubling or even unprofessional. (Since I am a consultant, I have lots of “employers,” and so far none of them seem to mind what I post. Some – the crazy ones – even enjoy my blog.)
Having a domain name owned by a weblog service: Pass. Since 1996 I have been keeping it real on moertel.com. My blog’s home is blog.moertel.com, which seems like the natural place for it.

In sum, I made three of Jakob’s top-ten weblog usability mistakes:

I don’t have an author bio.
I don’t have an author photo.
I don’t post regularly.

The first two are easy to fix, and I’ll fix them right away. The third – posting regularly – is more difficult, owing to the ever-varying demands of my work load, but I’ll make an effort to pick up the pace. Hopefully, my blog will 100 percent “Jakob compliant” in the next day or so.

Do you have a weblog? If so, how many weblog usability mistakes do you make? Grab Jakob’s top-ten list and find out.