Tom Moertel's Weblog: Database connection leak in Typo 4.0.3: problem solved

Database connection leak in Typo 4.0.3: problem solved

Tom Moertel — Thu, 24 Aug 2006 15:41:00 -0400

In an earlier post I wrote about stability problems that have plagued my blog since upgrading from Typo 4.0.0 to 4.0.3. I have finally traced the problem to its source, and here’s the deal:

If you’re serving Typo up via Mongrel, do not configure ActiveRecord to allow concurrency.

One of the changes between Typo 4.0.0 and 4.0.3 is this addition to the environment.rb file:

config.active_record.allow_concurrency = true

~~Comment out this line, restart Typo, and the problem is solved.~~ Apply Changeset 1255, and the problem is solved. (See Update 2, below.)

Discussion

When ActiveRecord::Base.allow_concurrency is set to true, AR will give each thread its own database connections and cache them in thread-localized storage. The idea is that, in a multi-threaded environment, this simple policy prevents unsafe interactions between threads and the database. (Imagine what would happen if one thread “borrowed” a connection over which another thread had opened a transaction. Oops, there goes transactional isolation.)

This policy, however, does place a burden on the owner of the threads to make sure that each thread’s local connection cache is cleared when the thread is joined, a burden that is not, it would seem, being carried by Typo under Mongrel. As a result, Typo rapidly chews through the allotment of file descriptors that the operating system kindly had reserved for Mongrel:

(On my Linux server, the Mongrel process gets an allotment of 1024 file descriptors.)

Lucky for us, this each-thread-gets-its-own-connections policy is unnecessary under Mongrel because Mongrel, while being multi-threaded itself, serializes all access to the Rails-based applications it serves up:

Q: Is [Mongrel] multi-threaded or can it handle concurrent requests?

Mongrel is uses a pool of thread workers to do it’s processing. This means that it is able to handle concurrent access and should be thread safe. This also means that you have to be more careful about how you use Mongrel. You can’t just write your application assuming that there are no threads involved. ...

Ruby on Rails is not thread safe so there is a synchronized block around the calls to Dispatcher.dispatch. This means that everything is threaded right before and right after Rails runs. While Rails is running there is only one controller in operation at a time.
(Source: Mongrel FAQ list)

Thus we can safely turn off (i.e., comment out in Typo’s environment.rb file) ActiveRecord’s allow-currency option without having to worry about nasty concurrency or performance issues:

# the following line is commented out
# config.active_record.allow_concurrency = true

For more on this subject, see Rails ticket #2162 and Rails ticket #2742.

Now, here’s my question: Are there any environments in which Typo can run with the allow-concurrency option enabled and not leak database connections? Inquiring minds want to know.

Update: Upon further investigation, turning off concurrency might not be altogether without risk. Some of the Typo code that handles potentially long tasks, such as making trackbacks and pings, spawns new threads in which to carry out its work. I’m looking further into this risk. Updates to come.

Update 2: Piers Cawley added Changeset 1255, which turns AR’s allow-concurrency flag back off and revises the ping code so that it does not attempt concurrent database access. Apply the patch version of 1255 and restart Typo to get the fix. A tip of the hat to Piers for making the quick fix when he was supposed to be on holiday.

"Database connection leak in Typo 4.0.3: problem solved" by Adam Byrtek

Tue, 05 Aug 2008 18:42:24 -0400

For future reference: this problem was caused by the fact that SQLite adapter in Rails didn’t handle closing of old connections properly. Using allow_concurrency only made this visible. The problem was fixed in this commit

"Database connection leak in Typo 4.0.3: problem solved" by Tim Harper

Thu, 03 Jul 2008 12:50:17 -0400

Well… since I just posted a comment I realized I was wrong about allow_concurrency making ActiveRecord thread safe.

http://tinyurl.com/6b4m6h

So, what is it good for? I have no idea now.

"Database connection leak in Typo 4.0.3: problem solved" by Tim Harper

Thu, 03 Jul 2008 11:22:47 -0400

Woah was I late to the party :)

"Database connection leak in Typo 4.0.3: problem solved" by Tim Harper

Thu, 03 Jul 2008 11:22:22 -0400

ActiveRecord.allow_concurrency = true is only good for making it thread safe. However, contrary to what you may believe (and perhaps depending on the database adapter you are using), it does not allow you to run queries in parallel (though it does allocate a connection for each thread.

This is because of ruby’s green threads which are processed by the VM. Everytime a native function is run, ruby seems to lose control of the execution and therefore can no longer process the running threads.

POSIX threads should fix this.

"Database connection leak in Typo 4.0.3: problem solved" by Piers Cawley

Fri, 25 Aug 2006 03:04:42 -0400

I think I need to get my brain fixed. r1255 is the one you want.

[Piers wanders off into the night, mumbling to himself and occasionally exlaiming “Millenium hand and shrimp!”]

"Database connection leak in Typo 4.0.3: problem solved" by Piers Cawley

Fri, 25 Aug 2006 02:58:46 -0400

Sorry, 1254

"Database connection leak in Typo 4.0.3: problem solved" by Piers Cawley

Fri, 25 Aug 2006 02:56:26 -0400

And, as the guilty party in turning concurrency back on, I woke up this morning, made Ping threadsafe (I think) and turned concurrency back off.

Revision 1245 on the trunk if you’re doing 4.1 stuff, but I’m sure it’ll get backported to the stable branch and you’ll find 4.04 getting released shortly thereafter.

"Database connection leak in Typo 4.0.3: problem solved" by Scott Laird

Thu, 24 Aug 2006 19:55:53 -0400

Oops—it looks like we screwed this up. I didn’t realize that we’d turned concurrency on. Unfortunatenly, turning it off breaks Postgres right now. Hmm.