Database connection leak in Typo 4.0.3: problem solved

Tom Moertel — Thu, 24 Aug 2006 15:41:00 -0400

In an earlier post I wrote about stability problems that have plagued my blog since upgrading from Typo 4.0.0 to 4.0.3. I have finally traced the problem to its source, and here’s the deal:

If you’re serving Typo up via Mongrel, do not configure ActiveRecord to allow concurrency.

One of the changes between Typo 4.0.0 and 4.0.3 is this addition to the environment.rb file:

config.active_record.allow_concurrency = true

~~Comment out this line, restart Typo, and the problem is solved.~~ Apply Changeset 1255, and the problem is solved. (See Update 2, below.)

Discussion

When ActiveRecord::Base.allow_concurrency is set to true, AR will give each thread its own database connections and cache them in thread-localized storage. The idea is that, in a multi-threaded environment, this simple policy prevents unsafe interactions between threads and the database. (Imagine what would happen if one thread “borrowed” a connection over which another thread had opened a transaction. Oops, there goes transactional isolation.)

This policy, however, does place a burden on the owner of the threads to make sure that each thread’s local connection cache is cleared when the thread is joined, a burden that is not, it would seem, being carried by Typo under Mongrel. As a result, Typo rapidly chews through the allotment of file descriptors that the operating system kindly had reserved for Mongrel:

(On my Linux server, the Mongrel process gets an allotment of 1024 file descriptors.)

Lucky for us, this each-thread-gets-its-own-connections policy is unnecessary under Mongrel because Mongrel, while being multi-threaded itself, serializes all access to the Rails-based applications it serves up:

Q: Is [Mongrel] multi-threaded or can it handle concurrent requests?

Mongrel is uses a pool of thread workers to do it’s processing. This means that it is able to handle concurrent access and should be thread safe. This also means that you have to be more careful about how you use Mongrel. You can’t just write your application assuming that there are no threads involved. ...

Ruby on Rails is not thread safe so there is a synchronized block around the calls to Dispatcher.dispatch. This means that everything is threaded right before and right after Rails runs. While Rails is running there is only one controller in operation at a time.
(Source: Mongrel FAQ list)

Thus we can safely turn off (i.e., comment out in Typo’s environment.rb file) ActiveRecord’s allow-currency option without having to worry about nasty concurrency or performance issues:

# the following line is commented out
# config.active_record.allow_concurrency = true

For more on this subject, see Rails ticket #2162 and Rails ticket #2742.

Now, here’s my question: Are there any environments in which Typo can run with the allow-concurrency option enabled and not leak database connections? Inquiring minds want to know.

Update: Upon further investigation, turning off concurrency might not be altogether without risk. Some of the Typo code that handles potentially long tasks, such as making trackbacks and pings, spawns new threads in which to carry out its work. I’m looking further into this risk. Updates to come.

Update 2: Piers Cawley added Changeset 1255, which turns AR’s allow-concurrency flag back off and revises the ping code so that it does not attempt concurrent database access. Apply the patch version of 1255 and restart Typo to get the fix. A tip of the hat to Piers for making the quick fix when he was supposed to be on holiday.

Cool stuff: Composable memory transactions

Tom Moertel — Sat, 09 Apr 2005 14:10:00 -0400

If you write software that deals with concurrency, you are doubtless familiar with the painful limitations of the concurrency abstractions that most programming languages, runtimes, and operating systems offer us humble programmers. The one-big-select idiom, the need to impose a global ordering policy on lock taking, and the myriad things that can unexpectedly bite you in the behind when managing threads are not-so-subtle reminders that the programming world still has a few fundamental problems to solve.

Thus I was impressed when I read Tim Harris, Simon Marlow, Simon Peyton Jones, and Maurice Herlihy’s paper on Composable memory transactions a couple of months ago. The paper introduced some Very Cool Stuff (especially if you program in Haskell, for which there is an implementation available). More recently at the Links meeting at ETAPS (another cool thing a’brewing), the same team gave a talk on the subject: Concurrency Unlocked: transactional memory for composable concurrency. Check out the slides from the talk for a summary of the problem and the STM solution.

The gist is that today’s ubiquitous concurrency abstraction – the lock – is fundamentally at odds with the most successful technique we humans have for building complex systems: gluing simple systems together. Composable memory transactions, on the other hand, do not have this problem. As a result, they offer a fundamentally simpler and more mentally scalable solution for building complex concurrent systems.

To quantify this coolness, we have only to look at section 4.2 of the paper. In about 25 lines of code the authors give an implementation for what is effectively the heart of an instant messaging server. I am not kidding. Multiple writers with serialized writes into each channel, multiple readers on each channel with independent read positions and buffering. Yeah, it’s in there.

Do yourself a favor. Check out the slides from the talk and then read the paper. This is some seriously cool stuff. You ought to know about it.

Update 2007-01-16: Since I wrote this article, Software Transactional Memory has received a lot of attenttion. Here are a couple of pointers worth checking out:

Beautiful Concurrency (PostScript paper). An approachable introduction to STM that doesn’t assume you already understand Haskell.
Wikipedia’s entry on Software Transactional Memory is pretty good, too.

Tom Moertel's Weblog: Tag concurrency

Database connection leak in Typo 4.0.3: problem solved

Discussion

Cool stuff: Composable memory transactions