Since I upgraded my blog from Typo 4.0.0 to 4.0.3, it has been somewhat unstable. About once a day it starts responding with “500 Internal Server Error” and stays that way until I restart it.
The root of the problem seems to be the database connection, as evidenced by this exception showing up in the production log:
SQLite3::CantOpenException (could not open database)
Unfortunately, the exception doesn’t provide anything specific to go on.
A quick look at the sqlite3-ruby code suggested that I was not going to get the specifics, either. The Ruby-based wrapper never calls sqlite3_errmsg after a call to sqlite3_open fails on behalf of SQLite3::Database.new.
A quick patch, however, fixed the problem:
--- sqlite3-ruby-1.1.0.orig/lib/sqlite3/database.rb
+++ sqlite3-ruby-1.1.0/lib/sqlite3/database.rb
@@ -109,7 +109,7 @@
@statement_factory = options[:statement_factory] || Statement
result, @handle = @driver.open( file_name, utf16 )- Error.check( result, nil, "could not open database" )
+ Error.check( result, self, "could not open database" )
@closed = false @results_as_hash = options.fetch(:results_as_hash,false)
(Submitted as Ticket 5504 on RubyForge)
Before applying the patch, opening a database at a nonexistent path results in a generic error message:
$ ruby -r rubygems -e 'require_gem "sqlite3-ruby";
SQLite3::Database.new("/no/such/path/db")'
... could not open database (SQLite3::CantOpenException) ...
After applying the patch, we get additional error information:
... could not open database: unable to open database file
(SQLite3::CantOpenException) ...
With the patch in place, all I have to do is wait for Typo to start acting up again. Then I’ll have some interesting information in the log.
Until then, I’m relying on cron and a short monitoring script to restart Typo when it tips into foolishness:
#!/bin/bash
url=http://blog.moertel.com/admin
addrs=tom@moertel.com
response=$(GET -sd $url 2>&1)
if [ "$response" != "200 OK" ]; then
{ echo "Response was: $response"; echo; service typo restart; } |
mail -s "Blog site not responding! (Restarting)" $addrs
fi
We’ll see how it goes.
Update: That was fast. The error popped up again and this time the log told me something useful: “unable to open database file.” Now, why couldn’t Typo open the database file, especially since the file is perfectly fine and had been opened successfully (many times) by the very same Typo process earlier? Here’s a hint:
$ ls /proc/28788/fd | wc -l
1023
Seems like there’s a resource leak in Typo 4.0.3 (or Rails 1.1.6).
Under some conditions, instead of reusing existing database
connections, Typo keeps trying to open new ones. Eventually, it uses
up its allotment of file descriptors and the operating system is forced
to say, “That’s enough, pal,”
(EMFILE
).
I’ll look in to it more in the morning.
Update 2: Problem solved.