Tom Moertel's Weblog

I have written a short novel's worth of content for my blog

Posted by Tom Moertel Fri, 30 Mar 2007 04:34:00 GMT

How much content have I written for my blog? Let’s find out.

My blog runs on Typo, which is built upon Ruby on Rails. Let’s fire up the Rails console and gather a quick word count:

$ cd ~/blog
$ ruby script/console 
Loading development environment.
>> require 'article'
=> true
>> Article.find(:all).inject(0) { |sum,a| sum +=
       (a.body + a.extended.to_s).split(/\s+/).length }
=> 66665

So I have written about 66 kilo-words, which is entering novel territory. Paperback-wise, it’s about 190 pages.

All I need now is a villain and some cool cover art.

Posted in site news, rails, writing
Tags blog, rails, word_count, words, writing
2 comments
no trackbacks
 

Ruby 1.9 gets handy new method Object#tap

Posted by Tom Moertel Wed, 07 Feb 2007 17:08:00 GMT

Via eigenclass.org I learned that Ruby 1.9 will sport a new Object method called tap, which is something I’ve been hoping for.

What’s tap? It’s a helper for call chaining. It passes its object into the given block and, after the block finishes, returns the object:

an_object.tap do |o|
  # do stuff with an_object, which is in o
end # ===> an_object

The benefit is that tap always returns the object it’s called on, even if the block returns some other result. Thus you can insert a tap block into the middle of an existing method pipleline without breaking the flow. MenTaLguY has some nifty examples of other things you can do with tap.

Fans of Ruby on Rails may recognize tap as similar to RoR’s own returning helper.

Looks like Ruby 1.9 is going to be extra cool for a number of reasons.

Posted in ruby, rails
Tags helpers, rails, ruby, tap
1 comment
no trackbacks
 

A type-based solution to the "strings problem": a fitting end to XSS and SQL-injection holes?

Posted by Tom Moertel Thu, 19 Oct 2006 01:40:00 GMT

Even skilled programmers have a hard time keeping their web applications free of XSS and SQL-injection vulnerabilities. And it shows: a sobering portion of web sites are open to some scary security threats.

Why are so many sites vulnerable to these well-known holes? Probably because it’s insanely hard for programmers to solve the fundamental “strings problem” at the heart of these vulnerabilities. The problem itself is easy to understand, but we humans aren’t equipped to carry out the solution. Simply put, we just plain suck at keeping a bazillion different strings straight in our heads, let alone consistently and reliably rendering their interactions safe whenever they cross paths in a modern web application. It’s easy to say, “just escape the little buggers,” but it’s hard to get it right, every single time.

Computers, on the other hand, are pretty good at keeping track of details by the bucket-full. Wouldn’t it be nice, then, if our programming languages gave us the power to delegate this nasty “strings problem” to our computers, which could then devote their unwavering mechanical precision to grinding the problem out of existence? Isn’t that the kind of thing modern programming languages are supposed to be good at?

I’d like to think the answer to that question is a big, you betcha.

So let’s grab a modern programming language and solve the strings problem.

Let’s solve the strings problem in Haskell

In this article, we will look at one way (among many) to solve the strings problem: by adding Ruby-style string templates to Haskell. These templates support “interpolation” via the usual, convenient #{var} syntax, but here interpolation is type safe. Haskell’s type system will prevent us from inadvertently mixing incompatible string types, and it will detect mistakes at compile time, before they can become live XSS or SQL-injection holes. Further, our solution will offer us these benefits without making us jump through hoops or pay some onerous syntax penalty.

To be more specific, the system offers the following benefits:

• It provides a string-management kernel that lets you create “safe strings” by certifying a regular string as representing either text or a fragment of a known language.
• It allows you to conveniently define new language types for any string-based language that you can provide an escaping rule for (e.g., XML, URLs, SQL, untrusted user input).
• It provides compile-time syntactic sugar (via Template Haskell) that makes working with safe strings as convenient as working with string interpolation in languages like Ruby and Perl.
• It catches and reports (at compile time) the following commonly made programming errors:
  • failing to escape a plain-old-text string before mixing it into a string that represents a language fragment
  • mixing strings that represent fragments of incompatible languages
  • mixing strings that represent fragments of compatible languages in an ambiguous way (the system will force you to disambiguate)

(This is a long one, so grab an espresso, lean back, and read on in style. Also, if you have a smoking jacket, you might want to get it now.)

Posted in programming, programming languages, haskell, ruby, web development, testing, rails
Tags haskell, ruby, strings, testing, types
37 comments
no trackbacks
 

If unit testing can't keep Rails safe from string-escaping problems, what makes you think it will keep your projects safe?

Posted by Tom Moertel Thu, 12 Oct 2006 20:06:00 GMT

Recently I wrote about unit testing being a tool, not a goal in itself. I argued that unit testing was not a reliable way to fight certain kinds of common coding errors and, therefore, that unit testing ought to be supplemented with other tools.

To support my argument, I gave an example of a common, important coding error that unit testing does a bad job of helping programmers control. That error is failing to manage and escape strings properly: the “strings problem.” It is the mother of XSS and SQL-injection security vulnerabilities, not to mention the cause of legions of broken links and bad HTML on the web.

If you think I’m overstating the problem, or if you think that unit testing is a good way of solving it, let me show you how easy it is for even smart developers to get it wrong.

Consider Ruby on Rails, a great framework for developing web applications. Rails has an extensive suite of unit tests, and the Rails development guidelines require that changes to Rails be accompanied by unit tests that “prove [the] change works.”

Now consider that one of Rails’s most-used and most-scrutinized methods – the venerable link_to helper – contains a fundamental string-escaping error:

require 'rubygems'
require_gem 'rails'
include ActionView::Helpers::UrlHelper

url = "http://example.com?ohms_law?volt=1&amp=3" 
puts link_to("TEST", url)

The code, when executed, prints the following HTML snippet:

<a href="http://example.com?ohms_law?volt=1&amp=3">TEST</a>

The HTML snippet represents a hypertext link. The link should point to the URL given in the code, but because the URL was not properly escaped when it was converted into HTML by the link_to helper, the link is broken:

CORRECT:  http://example.com?ohms_law?volt=1&amp=3
LINK_TO:  http://example.com?ohms_law?volt=1&=3
                                             ^ oops

Here’s what’s going on. Because the URL was not escaped, web browsers misinterpret its “amp” parameter as a character-entity reference, which gets gobbled up when the link’s href attribute is parsed. (To see this for yourself, save the output of the Ruby code into an HTML file, open the file with your favorite web browser, and see where the link points.)

Now, how come the unit tests didn’t catch this problem? It turns out, the tests got it wrong, too, by expecting broken output:

# in url_helper_test.rb

def test_link_tag_with_query
  assert_dom_equal \
    "<a href=\"http://www.example.com?q1=v1&amp;q2=v2\">Hello</a>",
    link_to("Hello", "http://www.example.com?q1=v1&amp;q2=v2")
end

The point isn’t that the Rails developers are dumb. The point is that the Rails developers are smart. If they can’t get the strings problem right, even with all their brains and all their unit testing, what reason does any programmer have to think that unit testing is going to solve this problem reliably?

If, then, you want to solve the strings problem – and you really, seriously ought to want to solve the strings problem – you should consider options beyond unit testing.

Posted in programming, testing, rails
Tags escaping, problem, rails, strings, testing
5 comments
no trackbacks
 

Database connection leak in Typo 4.0.3: problem solved

Posted by Tom Moertel Thu, 24 Aug 2006 19:41:00 GMT

In an earlier post I wrote about stability problems that have plagued my blog since upgrading from Typo 4.0.0 to 4.0.3. I have finally traced the problem to its source, and here’s the deal:

If you’re serving Typo up via Mongrel, do not configure ActiveRecord to allow concurrency.

One of the changes between Typo 4.0.0 and 4.0.3 is this addition to the environment.rb file:

config.active_record.allow_concurrency = true

Comment out this line, restart Typo, and the problem is solved. Apply Changeset 1255, and the problem is solved. (See Update 2, below.)

Discussion

When ActiveRecord::Base.allow_concurrency is set to true, AR will give each thread its own database connections and cache them in thread-localized storage. The idea is that, in a multi-threaded environment, this simple policy prevents unsafe interactions between threads and the database. (Imagine what would happen if one thread “borrowed” a connection over which another thread had opened a transaction. Oops, there goes transactional isolation.)

This policy, however, does place a burden on the owner of the threads to make sure that each thread’s local connection cache is cleared when the thread is joined, a burden that is not, it would seem, being carried by Typo under Mongrel. As a result, Typo rapidly chews through the allotment of file descriptors that the operating system kindly had reserved for Mongrel:

(On my Linux server, the Mongrel process gets an allotment of 1024 file descriptors.)

Lucky for us, this each-thread-gets-its-own-connections policy is unnecessary under Mongrel because Mongrel, while being multi-threaded itself, serializes all access to the Rails-based applications it serves up:

Q: Is [Mongrel] multi-threaded or can it handle concurrent requests?

Mongrel is uses a pool of thread workers to do it’s processing. This means that it is able to handle concurrent access and should be thread safe. This also means that you have to be more careful about how you use Mongrel. You can’t just write your application assuming that there are no threads involved. ...

Ruby on Rails is not thread safe so there is a synchronized block around the calls to Dispatcher.dispatch. This means that everything is threaded right before and right after Rails runs. While Rails is running there is only one controller in operation at a time.

(Source: Mongrel FAQ list)

# the following line is commented out
# config.active_record.allow_concurrency = true

For more on this subject, see Rails ticket #2162 and Rails ticket #2742.

Now, here’s my question: Are there any environments in which Typo can run with the allow-concurrency option enabled and not leak database connections? Inquiring minds want to know.

Posted in ruby, typo, rails
Tags activerecord, concurrency, rails, sqlite3, typo
4 comments
no trackbacks
 

Typo-4.0.3 instability and a minor patch for sqlite3-ruby

Posted by Tom Moertel Thu, 24 Aug 2006 04:41:00 GMT

Since I upgraded my blog from Typo 4.0.0 to 4.0.3, it has been somewhat unstable. About once a day it starts responding with “500 Internal Server Error” and stays that way until I restart it.

The root of the problem seems to be the database connection, as evidenced by this exception showing up in the production log:

SQLite3::CantOpenException (could not open database)

Unfortunately, the exception doesn’t provide anything specific to go on.

A quick look at the sqlite3-ruby code suggested that I was not going to get the specifics, either. The Ruby-based wrapper never calls sqlite3_errmsg after a call to sqlite3_open fails on behalf of SQLite3::Database.new.

A quick patch, however, fixed the problem:

--- sqlite3-ruby-1.1.0.orig/lib/sqlite3/database.rb
+++ sqlite3-ruby-1.1.0/lib/sqlite3/database.rb
@@ -109,7 +109,7 @@
       @statement_factory = options[:statement_factory] || Statement

       result, @handle = @driver.open( file_name, utf16 )
-      Error.check( result, nil, "could not open database" )
+      Error.check( result, self, "could not open database" )

       @closed = false
       @results_as_hash = options.fetch(:results_as_hash,false)

(Submitted as Ticket 5504 on RubyForge.)

Before applying the patch, opening a database at a nonexistent path results in a generic error message:

$ ruby -r rubygems -e 'require_gem "sqlite3-ruby";
    SQLite3::Database.new("/no/such/path/db")'

... could not open database (SQLite3::CantOpenException) ...

After applying the patch, we get additional error information:

... could not open database: unable to open database file
    (SQLite3::CantOpenException) ...

With the patch in place, all I have to do is wait for Typo to start acting up again. Then I’ll have some interesting information in the log.

Until then, I’m relying on cron and a short monitoring script to restart Typo when it tips into foolishness:

#!/bin/bash

url=http://blog.moertel.com/admin
addrs=tom@moertel.com

response=$(GET -sd $url 2>&1)

if [ "$response" != "200 OK" ]; then
    { echo "Response was: $response"; echo; service typo restart; } |
    mail -s "Blog site not responding! (Restarting)" $addrs
fi

We’ll see how it goes.

$ ls /proc/28788/fd | wc -l
1023

Posted in ruby, typo, rails, sysadmin
Tags rails, sqlite3, typo
1 comment
no trackbacks
 

The button_to helper is now part of Rails!

Posted by Tom Moertel Thu, 16 Jun 2005 16:00:00 GMT

I am delighted to report that the button_to helper has been added to the Ruby on Rails web-development framework. David applied the patch earlier today, and so button_to will be in the much-anticipated Rails 1.0 release.

David’s change-log entry summarizes the patch well:

Added button_to as a form-based solution to deal with harmful actions that should be hidden behind POSTs. This makes it just as easy as link_to to create a safe trigger for actions like destroy, although it’s limited by being a block element, the fixed look, and a no-no inside other forms.

David does a good job of highlighting the helper’s limitations. I’ll take this opportunity to elaborate on each.

It is a block element

The button_to helper creates a small form, which in HTML is considered block content, just like the p, div, and blockquote elements are. Basically, block content cannot be mixed into runs of text. But links can: links are inline content. Thus button_to cannot be used as a drop-in replacement for every occurrence of link_to that might be unsafe; it works only for those occurrences within block-accepting contexts.

Luckily for us, when designers use links to trigger unsafe actions, they rarely slip such links into the middle of ordinary looking text. Naughty uses of link_to almost always occur within contexts that accept block content. In Rails-generated scaffolding code, for instance, the unsafe uses of link_to occur within table cells, and table cells have a flow content model, which accepts both inline and block content. So button_to works great for the default cases in Rails.

It has a fixed look

As its name implies, button_to creates buttons. Buttons don’t look like links and aren’t styled the same way that links are. For some design scenarios, this might be a problem.

(My view is that links should not be used to trigger unsafe actions. In the same way that action-triggering GET requests violate the spirit of the HTTP standards, action-triggering hypertext links violate the spirit of the HTML standards. For this reason, I view this limitation as a feature.)

It is a no-no inside other forms

Forms cannot be nested, and so button_to cannot be used inside of forms.

Fortunately, this limitation usually doesn’t matter because when we are inside of a form, we can use its buttons instead of button_to-created buttons to trigger actions. Still, there are some circumstances where it does matter, such as the “Amazon.com wish list” scenario. In this scenario, we should consider other options.

The bottom line: Pick the low-hanging fruit

While button_to has its limitations, it does provide a simple solution to the unsafe-GET problem for most real-world cases. I am glad that it is now a part of Rails, and I offer a big thank-you to David for accepting the patch.

Posted in web development, rails
Tags get, gwa, link_to, post, rails, safe, unsafe
no comments
no trackbacks
 

Taking the unsafe GETs out of Rails

Posted by Tom Moertel Sun, 08 May 2005 16:00:00 GMT

As I wrote earlier, it’s time for web developers to do away with the fundamentally broken practice of using hypertext links to trigger dangerous events such as deleting things. One of the first places we ought to clean house is in the burgeoning Rails web-application framework, where this practice is pervasive.

The primary culprit in Rails is the all-too-easy link_to method, which is (presently) the orthodox means of creating links to any action, even unsafe ones. For example:

link_to "Destroy", :controller => 'accounts',
        :action => 'destroy', :id => 6

The above code generates the following HTML hypertext link, which when followed will merrily delete account number 6:

<a href="/accounts/destroy/6">Destroy</a>

Because this practice is dangerous and contrary to the decade-old convention that links be safe, the link_to method thoughtfully lets us request that a Javascript confirmation dialog be tacked onto the link for added protection:

link_to "Destroy", ...,  :confirm => "Are you sure?" 

The resulting “safe” HTML:

<a href="/accounts/destroy/6" 
   onclick="return confirm('Are you sure?');">Destroy</a>

Unfortunately, the Javascript protection doesn’t work. First, not all web browsers care about it. Lots of people surf with Javascript turned off. Second, a whole slew of things besides web browsers live on the Internet, and almost all of them are oblivious to Javascript. Web crawlers fall into this category. They will be more than happy to follow any link you feed to them. “Hey, Googlebot just deleted every account in our database!” Oops.

Thus another layer of protection is commonly used: authorization. The theory is that dangerous links can be safely corralled in the private parts of a web application, where the public and web crawlers cannot go. Only authorized users can get into those parts, and those users will be smart enough not to click on the truly dangerous links unless they really mean it.

The problem is, any number of intermediary agents can be operating on behalf of an authorized user, and these agents are free to do anything the user is allowed to do, such as follow dangerous links. Google’s Web Accelerator is one such agent. It tries to make your surfing faster by (among other things) pre-fetching the resources that are linked to on the pages you visit. And what happens if you, an authorized user, visit a page containing dangerous links? That’s right, Web Accelerator will fetch the “resources” those links point to – and delete a bunch of your stuff.

I hope by this point that I have argued convincingly that using links for unsafe actions is a bad idea. Even if you feel justified in ignoring the applicable parts of the HTTP RFCs, it’s a bad idea. Even if you tack on Javascript confirmations and hide your links in authorization-protected zones of your site, it’s a bad idea. It is, all around, a bad idea. Don’t do it.

So what alternatives are there? Read on for one possibility, button_to.

Posted in ruby, web development, rails
Tags get, gwa, link_to, post, rails, safe, unsafe
8 comments
no trackbacks