Typo-4.0.3 instability and a minor patch for sqlite3-ruby
Posted by Tom Moertel Thu, 24 Aug 2006 04:41:00 GMT
Since I upgraded my blog from Typo 4.0.0 to 4.0.3, it has been somewhat unstable. About once a day it starts responding with “500 Internal Server Error” and stays that way until I restart it.
The root of the problem seems to be the database connection, as evidenced by this exception showing up in the production log:
SQLite3::CantOpenException (could not open database)
Unfortunately, the exception doesn’t provide anything specific to go on.
A quick look at the sqlite3-ruby code suggested that I was not going to get the specifics, either. The Ruby-based wrapper never calls sqlite3_errmsg after a call to sqlite3_open fails on behalf of SQLite3::Database.new.
A quick patch, however, fixed the problem:
--- sqlite3-ruby-1.1.0.orig/lib/sqlite3/database.rb
+++ sqlite3-ruby-1.1.0/lib/sqlite3/database.rb
@@ -109,7 +109,7 @@
@statement_factory = options[:statement_factory] || Statement
result, @handle = @driver.open( file_name, utf16 )
- Error.check( result, nil, "could not open database" )
+ Error.check( result, self, "could not open database" )
@closed = false
@results_as_hash = options.fetch(:results_as_hash,false)
(Submitted as Ticket 5504 on RubyForge.)
Before applying the patch, opening a database at a nonexistent path results in a generic error message:
$ ruby -r rubygems -e 'require_gem "sqlite3-ruby";
SQLite3::Database.new("/no/such/path/db")'
... could not open database (SQLite3::CantOpenException) ...
After applying the patch, we get additional error information:
... could not open database: unable to open database file
(SQLite3::CantOpenException) ...
With the patch in place, all I have to do is wait for Typo to start acting up again. Then I’ll have some interesting information in the log.
Until then, I’m relying on cron and a short monitoring script to restart Typo when it tips into foolishness:
#!/bin/bash
url=http://blog.moertel.com/admin
addrs=tom@moertel.com
response=$(GET -sd $url 2>&1)
if [ "$response" != "200 OK" ]; then
{ echo "Response was: $response"; echo; service typo restart; } |
mail -s "Blog site not responding! (Restarting)" $addrs
fi
We’ll see how it goes.
$ ls /proc/28788/fd | wc -l
1023
Seems like there’s a resource leak in Typo 4.0.3 (or Rails 1.1.6).
Under some conditions, instead of reusing existing database
connections, Typo keeps trying to open new ones. Eventually, it uses
up its allotment of file descriptors and the operating system is forced
to say, “That’s enough, pal,” (EMFILE).
I’ll look in to it more in the morning.
Update 2: Problem solved.
readers
Ironically, the first time I tried to comment this blog was down (after finding this in the feed).
I have a cron that runs touch on the public/dispatch every couple of hours. 4.0.2 seemed more stable (because it’s before they started heavy refactoring).
They probably need more comprehensive test coverage (including performance tests) before they do any more refactoring.