Tom Moertel's Weblog: Tag statistics

R tips and tricks: Producing smooth bitmap plots

Tom Moertel — Sat, 25 Aug 2007 21:56:00 -0400

The R statistics system can produce first-class data visualizations, commonly known as plots. Internally, plots are represented in an abstract graphics format that can be rendered on any of R’s wide range of graphics “devices” to produce concrete output – windows, bitmap files, PostScript files, PDF files, and others.

The bitmap formats, such as PNG, are preferred for posting plots online because of their widespread support by web browsers. The default bitmap-rendering devices in R, unfortunately, produce graphics that look a little too “bitmapped” for modern web tastes. Here, for example, is a plot rendered by R’s “png” device:

There’s nothing technically wrong with the plot, but it looks out of place on a web page. That’s because modern web browsers use font-smoothing and anti-aliasing techniques to render just about everything else on the page. Against this clean, un-jagged backdrop, the oh-so-bitmapped plot looks like a throwback to a previous era.

Happily, we can produce clean, anti-aliased R plots with a little help. Here’s the earlier plot, anti-aliased:

To produce the anti-aliased plot, I used R to produce a PDF file. Then I rendered the PDF file into a PNG image at 300 dpi using Ghostscript. Finally, I scaled the 300-dpi image down to screen resolution, producing a high-quality, anti-aliased result.

Here’s the recipe in detail.

First, I define an R function called pdfit that takes an abstract graphics object and makes a PDF-file rendering of it, using my preferred graphics-device settings:

require("lattice")

pdfit <- function(f, ...) {
  trellis.device(dev=pdf, theme="col.whitebg", ...);
  print(f);
  dev.off()
}

Then, when I create a plot I want to publish, I use pdfit to render it into a PDF file:

P.img <- xyplot( subs.low + subs.high ~ date, ... )

pdfit(P.img, file="image-downloads.pdf")  # render plot into PDF file

Finally, I use Ghostscript and ImageMagick to convert the PDF file into a high-quality, anti-aliased PNG file. (I keep both formats: the PDF file is best for publishing in printed papers, and the PNG file is best for posting online.) I use a simple Makefile to automate the process of converting the PDF files into PNG files:

# Makefile (GNU make)

pdfs := $(wildcard *.pdf)
pngs := $(pdfs:.pdf=.png)

all: $(pngs)
.PHONY: all

%.png: %.pdf
    gs -dSAFTER -dBATCH -dNOPAUSE -sDEVICE=png16m \
       -dGraphicsAlphaBits=4 -dTextAlphaBits=4 -r300 \
       -dBackgroundColor='16#ffffff' \
       -sOutputFile=$@ > /dev/null \
       $< && \
    mogrify -resize 500 $@

With this Makefile in my graphics directory, just a single “make” command is all it takes to convert my PDF images into anti-aliased PNG files, ready to post online.

And that’s it.

Do you have any tips or tricks for making good-looking graphics with R? If so, please do share.

Update: There is one downside to the sexy, anti-aliased plots: they are not as compressible as the old-style jagged plots. For the images above, for instance, the anti-aliased PNG file weighs in at 45 KB, but the original PNG file is a feathery 4.7 KB. So, if bandwidth is precious to you – or you’re planning on getting Slashdotted – you might want to stick with the jaggies.

Fun with statistics: estimating blog readership (a do-it-yourself recipe)

Tom Moertel — Wed, 22 Aug 2007 21:34:00 -0400

As everybody knows, statistics is fun. Is there anything cooler than crushing a heap of seemingly uninteresting numbers into gleaming jewels of meaning? Of course not! Models, data-visualization plots, and fat data sets are way cool. So, let’s find an excuse to play with them.

Here’s an excuse – I mean, an important and highly relevant question that many of us share: How many people actually read our blogs? To answer the question, we will need to use statistics, data, and cool plots. Further, if you’ve got the raw data for your blog, you can follow along with your own analysis. Even more fun!

We’ll start with a simple inspection of common web-log data, using command-line tools. After developing a rough understanding of what useful information we can extract, we’ll analyze the raw data using a series of successively more sophisticated techniques. In the end, we will derive a simple formula for estimating readership from easily obtainable data.

Sound good? Then let’s get rocking.

But first, a preemptive strike on would-be poo-pooers: I know all about FeedBurner. I know they will track my blog’s subscribers and use their mystical powers to infer the number of “real” subscribers I have. I know it’s all so easy. But easy isn’t the point. I want to understand what’s going on. Just taking somebody’s word for it isn’t nearly as satisfying as figuring it out yourself – nor as fun.

OK. For real this time, let’s get rocking.

The goal

We want to know how many people read my blog regularly. By regularly, I mean that if I post something today, we want to count the people who will read it within a week’s time. That way we’ll count the weekend readers but not the one-time readers who will trickle in from search engines over the months ahead.

We can’t just look at my web-log stats to determine my blog’s readership, however. That’s because a lot of people read my blog through online feed aggregators, such as Bloglines and Google Reader, and never actually “hit” my blog when they read it. (My blog is so ugly, in fact, that I would expect lots of my readers to use a feed aggregator just to protect themselves from my design “skills.”)

So the goal is to figure out how to count my readers using the data we can actually get our hands on.

The data

Here’s what we have: my HTTP server’s log. That’s it. Can we squeeze the good stuff from it? Let’s find out.

Each entry in the log represents a single request for something on my site. A typical entry looks like this (split over multiple lines for your reading pleasure):

72.14.199.81 - - [19/Aug/2007:19:31:43 -0400]
"GET /xml/atom/article/472/feed.xml HTTP/1.1" 200 1959 "-" 
"Feedfetcher-Google; (+http://www.google.com/...; 1 subscribers; ...)"

There’s a lot of potentially useful information in there:

the IP address of the host that made the request
the date and time that the request was received
a summary of the request (e.g., “GET /xml/atom/article/472/feed.xml HTTP/1.1”)
the response code, typically 200 for a successful response
the string sent by the requester’s user agent to identify itself (e.g., “Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; ...)”)

Note that this particular request was made by Google’s Feedfetcher for an Atom feed. Also note that Feedfetcher told us, via its user-agent identification string, how many of its users have subscribed to this particular feed. That’s good stuff we can use.

My blog’s main Atom feed is at /xml/atom10/feed.xml. There are other “main” feeds as well (e.g., RSS), but let’s focus on this one for now. Let’s see who’s been asking for it recently. First, I’ll create a bash-shell function to grab the subset of the log corresponding to 19 August:

$ get_subset() {
    fgrep "GET /xml/atom10/feed.xml" blog_log |
    fgrep 19/Aug/2007;
  }

Then I’ll summarize the user-agent part of that subset’s log entries:

$ get_subset |
  perl -lne 'print $1 if /"([^";(]+)[^"]*"$/' |
  sort | uniq -c | sort -rn

78 NewsGatorOnline/2.0
47 Vienna/2.1.3.2111
38 Mozilla/5.0
27 YandexBlog/0.99.101
21 NewsFire/69
20 Planet Haskell +http://planet.haskell.org/ ...
19 Feedfetcher-Google
19 AppleSyndication/54
14 Zhuaxia.com 1 Subscribers
14 NetNewsWire/2.1b33
13 RssFwd
13 Bloglines/3.1
11 livedoor FeedFetcher/0.01
10 Feeds2.0
 8 RssBandit/1.5.0.10
 8 Akregator/1.2.6
 7 Eldono
 6 Netvibes
 4 NetNewsWire/3.0
 2 trawlr.com
 2 Opera/9.21
 2 NetNewsWire/3.1b5
 2 NetNewsWire/2.1
 2 Mozilla/3.0
 1 Vienna/2.2.0.2206
 1 Vienna/2.1.0.2107
 1 NetNewsWire/2.1.1
 1 Liferea/1.2.10
 1 JetBrains Omea Reader 2.2
 1 FeedTools/0.2.26 +http://www.sporkmonger.com/projects/feedtools/
 1 Feedshow/2.0

Of the user agents that fetched my feed, only some, such as Bloglines and Google Reader, aggregate on behalf of other users, and only some of those mass aggregators reported how many people have subscribed through them:

$ get_subset |
  perl -lne 'print $1 if /"([^"]*?\d+ subscribers?)/i' |
  sort | uniq -c | sort -rn

78 NewsGatorOnline/2.0 (... 22 subscribers
19 Feedfetcher-Google; (... 102 subscribers
14 Zhuaxia.com 1 Subscribers
13 RssFwd (... 1 subscribers
13 Bloglines/3.1 (http://www.bloglines.com; 82 subscribers
11 livedoor FeedFetcher/0.01 (... 1 subscriber
10 Mozilla/5.0 (Rojo 1.0; ... 4 subscriber
 7 Eldono (http://www.eldono.de; 1 subscribers
 6 Netvibes (http://www.netvibes.com/; 12 subscribers
 2 trawlr.com (+http://www.trawlr.com; 4 subscribers
 1 Feedshow/2.0 (http://www.feedshow.com; 1 subscriber

Of the user agents that don’t report subscriber counts, most are single-user feed readers. The 47 requests from the Vienna-2.1.3.2111 reader, for example, came from 5 distinct IP addresses (which I’ve obscured to protect my innocent readers’ identities):

$ get_subset |
  perl -lane 'print $F[0] if m{"Vienna/2.1.3.2111}' |
  sort | uniq -c | sort -rn

22 121.44.xxx.xxx
20 208.120.xxx.xxx
 3 69.154.xxx.xxx
 1 84.163.xxx.xxx
 1 202.89.xxx.xxx

Does that mean I have only 5 distinct readers using Vienna 2.1.3.2111? Not necessarily. The first IP address, for example, could represent a firewall that serves several people from a single corporate campus. So there could, indeed, be more than 5 users lurking behind those addresses, but it’s hard to know for sure.

Thus we can’t rely on feed-fetching statistics to reliably determine the count of readers. The mass aggregators don’t all report their subscriber counts, and the stand-alone aggregators’ fetching habits are not readily interpreted. And, even if we could obtain reliable fetching inferences, that only tells us how many people fetched my blog’s feeds. We want to know how many people read my blog – actually look at the articles.

To do that, we’ll need a more-sophisticated approach.

A different approach: counting image downloads

Every once in a while, I’ll post an article that contains photos or graphs of something I’m trying to explain. Since images like that are included by reference, they are not actually part of the article itself. So when a feed fetcher grabs a syndicated copy of the article, it won’t bother to fetch the images. There’s no need to use the bandwidth unless the person on the other side of the feed actually reads the article, at which time the person’s feed reader can download the images on demand.

Thus we can use the number of image downloads as an estimate of the number of people who actually read my blog. For each article that has images, we can count how many times each image was downloaded during the article’s first week online and take the average of the counts as an estimate of the number of people who read the article. (Marketing weasels use this technique, too, to track your reading habits. The only difference is that they will often insert gratuitous, personally identifying images – web bugs – into their documents to track you specifically.)

The image-counting technique isn’t foolproof, however. Requests from people behind proxy servers may never actually make it to my server to be counted, leading to under-counting. Also, some web crawlers fetch images, which may artificially inflate the count of “readers.” Examining the logs, I didn’t see many image requests from crawlers, so our primary concern is under-counting. Since I’m OK with a conservative count, under-counting is acceptable.

Let’s give image-counting a try. On 15 July 2007, I posted a story about a nasty hailstorm that hit my neighborhood. The story included some photos of the storm and its aftermath. Let’s count how many times the second photo in the story was requested on the day the story was posted:

$ fgrep "webcam-2007-07-13--153421.jpg" mc_log |
  fgrep 15/Jul/2007 | wc -l

884 times. Many of those downloads, however, were made by just a few requesting hosts. Here are the top ten downloaders:

$ fgrep "webcam-2007-07-13--153421.jpg" mc_log |
  fgrep 15/Jul/2007 |
  perl -lane 'print $F[0]' |
  sort | uniq -c | head


 42 84.45.xxx.xxx
 27 192.168.xxx.xxx
 10 75.182.xxx.xxx
  7 83.132.xxx.xxx
  7 213.203.xxx.xxx
  6 72.173.xxx.xxx
  6 67.180.xxx.xxx
  6 65.214.xxx.xxx
  5 89.98.xxx.xxx
  5 85.104.xxx.xxx

How do we interpret these duplicate requests? One way would be to say that each request, duplicate or not, represents a unique reader. It’s plausible. When many readers share a gateway firewall, say in a corporate setting, they will all end up making requests from the same IP address(es). Thus, if we want to count all such readers, we should count all of the requests.

The more conservative interpretation is that all of the requests from the same IP address represent only a single reader. All of the duplicate requests might be reloads or, perhaps, the work of an overzealous user-agent working (inefficiently) on behalf of that user. Let’s recount using this conservative assumption:

$ fgrep "webcam-2007-07-13--153421.jpg" mc_log |
  fgrep 15/Jul/2007 |
  perl -lane 'print $F[0]' | sort -u | wc -l

So what’s the real count, 635 or 884? The truth probably lies somewhere in between. To make sure we capture the truth, then, let’s use both interpretations in our ongoing analysis. We will develop low and high estimates from now on.

If you have sharp eyes, you may have noticed that the second IP address in the list above was from a private network. That address, in fact, belongs to my workstation. When I write articles, I frequently reload the drafts, and reloading causes the images within the drafts to be re-fetched. We’ll need to filter out my addresses during our later analyses.

There’s one more thing to consider. We still need to count the image downloads for the rest of the week. So far, we have only counted those for the article’s first day online. So, let’s re-do our conservative count, only this time for the whole week. Let’s also filter out my private addresses and ignore all but HTTP 200 “OK” responses:

$ fgrep "webcam-2007-07-13--153421.jpg" mc_log |
  fgrep " 200 " |  # only count full downloads (status code = 200)
  grep -P '(1[56789]|2[01])/Jul/2007' |  # Jul 15 thru 21 (7 days)
  perl -lane 'print $F[0] unless $F[0] =~ /^192\.168\./' |
  sort -u | wc -l

So, we estimate conservatively that my article on the hailstorm was read by about 1600 people in its first week. Since the article was published on 15 July 2007, we can conservatively estimate that my blog’s regular readership was about 1600 at that time, too.

But that’s just a single point estimate. We’ll need more data if we’re to draw reliable conclusions.

Compiling the image data

To compile enough data for meaningful inferences, I have whipped up a small script (in Perl) to extract and summarize image-download statistics, given an HTTP-server log. Running the script on my blog’s log, here’s what we get:

Date	Hits low	Hits high	Image
2006-06-18	158	192	lady-beetle-larva-upside-down-small.jpg
2006-06-18	157	191	lady-battle-larva-upside-down-close.jpg
2006-07-06	163	200	lectro-shirt-before-and-after-wash-small
2006-07-06	163	203	lectro-shirt-before-wash-300dpi.jpg
2006-07-06	168	206	lectro-shirt-before-wash-small.jpg
2006-07-07	155	194	Cladonia-cristatella-close.jpg
2006-07-07	155	194	Cladonia-cristatella.jpg
2006-08-03	147	188	annies-mixup-0003.jpg
2006-08-03	146	188	annies-mixup-0002.jpg
2006-08-24	173	217	blog-fd-usage-vs-time.png
2006-09-12	271	328	perl-at-work-sign.png
2006-10-18	1448	1582	safe-strings.png *
2006-11-04	1005	1351	old-web-site-3.png
2006-11-04	1011	1364	old-web-site.png
2006-11-14	1265	1747	toms-apple-pie.jpg
2007-05-25	1567	2406	problem-close.jpg
2007-05-25	1563	2400	receiver-insides.jpg
2007-05-25	1551	2383	repair.jpg
2007-06-21	2290	3024	perl-and-r.png *
2007-07-15	1574	2360	webcam-2007-07-13—153751.jpg
2007-07-15	1562	2379	backyard-ice.jpg
2007-07-15	1553	2364	shredded.jpg
2007-07-15	1567	2346	webcam-2007-07-13—153757.jpg
2007-07-15	1561	2355	webcam-2007-07-13—153808.jpg
2007-07-15	1612	2469	hailstorm2.jpg
2007-07-15	1592	2382	webcam-2007-07-13—153726.jpg
2007-07-15	1586	2381	webcam-2007-07-13—153747.jpg
2007-07-15	1601	2404	webcam-2007-07-13—153421.jpg

Like most data sets, this one looks better in graphical form:

The circles represent our conservative readership estimates, and the pluses represent our liberal readership estimates. To interpret the overall readership trend, focus on one set of estimates, either circles or pluses.

What do we see? First, it looks like the quantity of downloads has increased steadily, from a few hundred in July 2006 to the low thousands by July 2007. That’s nice.

Second, the data are sparse. I don’t post images often, so we don’t have much data to go on.

Third, it looks like we have some outliers. If you look at the points near October 2006 and June 2007, you’ll see that they jump up from the surrounding points. (In the lower-bound series, I have marked these outliers with a short orange, vertical line segment.) If these jumps truly represented a sudden increase in readership, we would expect them to be permanent, reflected in later readership data. What we see, however, is that these gains are only temporary.

Thus it seems reasonable to conclude that something else is going on for these images. If you look back at the data table, I have marked the pair of curious images with asterisks. As it turns out, both of these images were part of stories that were featured on Reddit. So, what these data reflect is the normal readership plus the Reddit effect. To avoid throwing off our inferences, let’s discard the data for these two images.

In the end, we have a pretty good means of estimating my blog’s readership on the dates when I posted articles that contained images (provided those wily Redditers didn’t pile on the articles). The problem is, I would like to know what my readership is all the time, not just on those rare occasions I post images. I certainly don’t want to resort to using web bugs. Hey, I’m no marketing weasel.

It’s time to add yet another layer of sophistication to our analysis.

A combined model: reported subscribers with image downloads

Let’s go back to the subscriber numbers reported by online aggregators such as Bloglines and Google Reader. If we assume that those aggregators represent a decent slice of my readers, and that the size of that slice as a proportion of the whole universe of readers doesn’t change much over time, we can model actual readership (as gathered from image downloads) in terms of reported subscriber numbers. Then, we can use that model to predict actual readership for the dates when no image-download data are available.

That’s the plan. So, let’s get going.

Gathering subscriber data

So, let’s grab those subscriber numbers. Again, I’ve whipped up a Perl script to gather the data. Here’s what the script does. It –

scans my blog’s HTTP server log
ignores requests from private networks
ignores requests that don’t report a subscriber count
emits one subscriber count for each day of data in the log, computed as the sum of each feed’s subscriber count, as reported by each aggregator (if an aggregator fetches a feed more than once in a day, all but the final request are ignored)

Running the script on my server log, I got a large data set. It’s so large that I’ll go straight to the plot:

As you would expect, these subscriber counts are less than the corresponding reader counts we gathered from image downloads. Not everybody uses an online feed reader, after all.

One thing that leaps out is the discontinuity around February 2007. What happened back then? As it turns out, that is when Google finally started reporting its subscriber counts. Since Google has a large share of the online aggregator market, that one little change resulted in a big increase in the total of reported counts.

Still, that jump is going to make our analysis a bit more difficult. When we relate subscriber counts to actual readers, we will need to account for the “Google effect.”

Likewise, there are a few other sets of outliers – points that look like bogus data – we should keep in mind. To see whether any of our image-download data coincide with these outliers, let’s highlight our subscriber data for the days when we also have image data:

Sure enough, some of our early download data coincide with an outlier group in July 2006. Let’s remove that download data from our analysis set, too.

Our data cleaned, let’s move on.

The model

Now we are ready to relate subscribers to readers (as determined by downloads). Here’s our model:

y_i = a · g_i · x_i + e_i

Where:

y represents actual readers (as estimated from image downloads)
x represents subscribers as reported by online aggregators
i ranges over 1–N for our N data points
a is the coefficient that relates readers to subscribers
g is a true/false factor to indicate whether x includes Google Reader users
e is the model’s error term

What the model says is that readership (y) varies linearly with subscriber counts (x) and that the rate at which it varies is given by a · g_i. (Model aficionados may note that this is a varying-slope model.) The model does not include a constant term; this is to fix the y-intercept at 0 because when we have no actual readers, we cannot have any subscribers, either. Thus we know the point (0,0) must be part of the fitted model.

Here’s the data set we will use to fit our model:

        date y.low y.high   x     g
1 2006-06-18   158    192  53 FALSE
2 2006-08-03   146    188  68 FALSE
3 2006-08-24   173    217  89 FALSE
4 2006-09-12   271    328  97 FALSE
5 2006-11-04  1008   1358 112 FALSE
6 2006-11-14  1265   1747 114 FALSE
7 2007-05-25  1560   2396 385  TRUE
8 2007-07-15  1579   2382 401  TRUE

This data set combines a summarized version of our image-download data set with the corresponding data from our aggregator-reported subscriber set (the red points in the previous plot).

The low and high y values represent our conservative and liberal interpretations of readership, which we discussed earlier. You’ll also note that where multiple images were available for any particular date, I have averaged their download counts to give a centralized readership estimate for that date. (Exercise: For this model, why shouldn’t we include multiple images for a single date?)

Let’s plot this data set (just the y.low part):

There aren’t many points to go on, but because our model is so simple, there are probably enough. That means it’s time to fit our model to our data.

To fit our linear model, I’ll use the lm function from the amazingly cool R statistics system (which I’ve also been using for our plots). To summarize the results, I’ll use the display function from the “arm” CRAN package, which accompanies Andrew Gelman and Jennifer Hill’s wonderful book Data Analysis Using Regression and Multilevel/Hierarchical Models. (BTW, Gelman’s blog is fascinating. It’s one of my favorite reads.) If you are following along and don’t have the “arm” package installed, you can use the summary function instead of display.

First, let’s fit the model to the conservative data:

M1.low <- lm (y.low ~ g:x + 0, data=subs.readers)
display(M1.low)

lm(formula = y.low ~ g:x + 0, data = subs.readers)
         coef.est coef.se
gFALSE:x 6.31     1.59
gTRUE:x  3.99     0.64
  n = 8, k = 2
  residual sd = 356.94, R-Squared = 0.90

That’s a pretty good fit. Both of our model parameters are significant (even at the 1-percent level). The resulting model says that each subscriber represents about 4 actual readers (or 6.3 readers if the subscriber count doesn’t include Google Reader users).

Let’s visualize the model, now fit to our data:

The gray line segments represent our fitted model’s predictions. Thus, for example, when we have x = 100 reported subscribers, the model predicts that we have about y = 630 actual readers. Likewise, when we have 400 subscribers, the model predicts that we have about 1600 actual readers.

The two line segments show how our model accommodates the “Google effect.” On the left, we have the pre-Google slope; on the right, the post-Google slope. In effect, our model combines two simpler models and chooses between them based on the Boolean factor g.

And that’s all there is to the fitting process. Let’s repeat the process for the liberal-interpretation data.

M1.high <- lm (y.high ~ g:x + 0, data=subs.readers)
display(M1.high)

lm(formula = y.high ~ g:x + 0, data = subs.readers)
         coef.est coef.se
gFALSE:x 8.46     2.25
gTRUE:x  6.08     0.91
  n = 8, k = 2
  residual sd = 504.22, R-Squared = 0.91

Under this model, each subscriber represents about 6 actual readers (or 8.5 if our subscriber count doesn’t include Google Reader users).

Now that we have our models, let’s use them to predict actual readership.

Using our models for prediction

Models ready, we can now predict my blog’s readership for any day, not just those days on which I happened to include images in my postings.

I have subscriber data in an R data frame called, unsurprisingly, subscriber.data. It provides, for each day I have subscriber statistics, values for x and g. (This is the same data set visualized in the earlier plot “Aggregator-reported subscribers to blog.moertel.com.”) We can tell R to plug these values into our model to predict the actual number of readers for those days. Let’s make both conservative and liberal predictions, storing them in a new data frame called predicted.readers:

predicted.readers <-
  transform(subscriber.data,
            readers.low  = predict(M1.low, subscriber.data),
            readers.high = predict(M1.high, subscriber.data))

Now let’s plot our predictions. First the plot code, just so you can see how it’s done in R:

xyplot(readers.low + readers.high ~ date,
       data = predicted.readers,
       main = "Predicted actual readers of blog.moertel.com",
       ylab = "Readers",
       xlab = "Date",
       auto.key = list(x = .35, y = .9, corner = c(0,0),
                       text = c("conservative estimate",
                                "liberal estimate"),
                       reverse.rows = T, between = -19))

And the resulting plot:

The bottom line

We have distilled a ton of raw data into a simple formula for predicting my blog’s actual readership from readily available subscriber counts. Just take the total subscriber count and multiply by 4 and 6, respectively, for low and high estimates of readership.

So, to answer our original question, how many readers does my blog have? Only a few days ago, on 18 August, the online aggregators reported that they were serving my feeds to 442 subscribers. So we can predict that, right now, my blog has 1750 to 2650 readers.

We have our answer. Getting it took some doing, but the doing was fun, so all’s good.

Certainly, we could go on. There are many interesting questions left to be answered. What, for example, is the growth trend of my readership? What is Google Reader’s market share? For now, however, it’s time to take a break.

I hope you had fun following along. If you have your own data, I’d be interested in hearing about your analytical explorations. (And, if you haven’t installed R on your computer yet, do it now. R is seriously cool and comes with great documentation, examples, and sample data. If you’re not using R, you’re not having all the fun you deserve.)

Update: minor editing tweaks for clarity.

Greasmonkey script annotates IMDb movies with their decoder-ring percentile ranks

Tom Moertel — Wed, 11 Jul 2007 13:49:00 -0400

Sam at rephase.net has harnessed the earth-shattering power of the IMDb movie-rating decoder ring to create a Greasmonkey script that annotates IMDb-listed movies with their percentile ranks. Now you don’t need to look up a movie’s “star rating” in the decoder ring to see where the movie ranks; the ranking appears right on the movie’s IMDb page.

Do check out the script itself to see how Sam cleverly embeds a copy of the decoder ring and plucks scores from it as needed.

For more on the IMDb movie-rating decoder ring, see:

Talk: Fun with Numbers: R and Perl (and IMDB data)

Tom Moertel — Thu, 21 Jun 2007 14:38:00 -0400

Last week I gave a talk on the R statistics system and Perl for the Pittsburgh Perl Mongers. The example that threaded through the talk was something I have written about here before, extracting useful information from the Internet Movie Database. If you’ve read my earlier blog post or have used the Grand Unified IMDB Movie Rating Decoder Ring, you might find the slides from the talk interesting. They provide some more details about the R and Perl code used to analyze the IMDB data and create the decoder ring.

You can get the slides here:

New Fedora Core RPMS for CRAN packages arm, Matrix, lme4, car, coda, leaps, and mlmRev

Tom Moertel — Wed, 25 Apr 2007 14:07:00 -0400

Just a quick note for folks using the R statistics system on Fedora Linux. I have packaged for Fedora a bunch of R packages from the CRAN. (R packages have to be packaged again, as RPM packages, to integrate with Fedora Linux.)

My initial goal was to package arm, which contains tools for working with various regression models. (This package accompanies Andrew Gelman and Jennifer Hill’s wonderful book Data Analysis Using Regression and Multilevel/Hierarchical Models.) Packaging “arm,” however, quickly snowballed into packaging a bunch of prerequisites. Thankfully, I have now completed that task and can share the fruits of my labor with you.

All in all, to install “arm,” you will need the following RPMs:

R-arm-1.0-2
R-car-1.2-1
R-lme4-0.9975-1
R-Matrix-0.9975-1
R-R2WinBUGS-2.0-1

The following RPMs are optional (but you will need them if you want to rebuild the RPMs):

R-coda-0.10-1
R-leaps-2.7-1
R-mlmRev-0.995-1

You can download the packages from the RPMs section of the Community Projects site. Better yet, you can use Yum to download them for you. Just add the moertel-community Yum repository to your /etc/yum.repos.d directory (see RPMs for the recipe) and then use the following command:

$ sudo yum install R-arm

Yum will automatically resolve dependencies and install the required packages. If you want any of the optional packages, add them after “R-arm” on the command line.

I have built the packages for Fedora Core 6 on the x86_64 architecture, but the RPM specs are available if you want to rebuild the packages for other architectures. (See the instructions for rebuilding RPMs for help.)

Caveat: I’m not sure that the R-R2WinBUGS package is fully functional. It depends on BRugs, which doesn’t yet build on the Linux platform. To get around this problem, I made R-R2WinBUGS’s dependency on BRugs weak; the first package no longer requires the second to install.

Engauge Digitizer: a handy tool for extracting data from charts

Tom Moertel — Tue, 17 Apr 2007 03:45:00 -0400

Today I wanted to extract the data that were visualized in a chart I saw on Seth Roberts’s blog. That is, I had a picture of a data set, and I wanted the numbers behind the picture.

This task turned out to be surprisingly easy – once I found Engauge Digitizer, an open-source (GPL) tool made for this very task. After I launched Engauge, the digitization process was straightforward:

I established the chart’s coordinate system by clicking in the corners and entering the associated coordinates.
Then I had Engauge identify data points. With the mouse, I selected a data point by hand, teaching Engauge what a point looks like. Then Engauge identified spots on chart that looked like data points and locked on to them. I was able to step through the points to tell Engauge to skip the few it misidentified.
I manually selected a few more data points that were scrunched into blobs and had eluded Engauge’s point-detection heuristics.
Finally, I exported the data set in CSV format.

If you ever need to extract the data behind a chart, do check out Engauge Digitizer. (If you use Fedora Linux, you’ll be happy to know that I have packaged Engauge for you. Get it at the RPMs section of the community site.)

The IMDB Movie Rating Decoder Ring: updated w/ 2 March 2007 data

Tom Moertel — Fri, 09 Mar 2007 17:40:00 -0500

If you want to get more out of IMDB movie ratings, check out my IMDB Movie Rating Decoder Ring, now updated with fresher data (as of 2 March 2007).

Mining gold from the Internet Movie Database, part 1: decoding user ratings

Tom Moertel — Tue, 17 Jan 2006 20:59:00 -0500

The Internet Movie Database (IMDb) is a rich source of online movie information. The problem is, the true gold is buried deep beneath the site’s user-friendly exterior and hidden within the database itself. With a little digging, however, we can extract the gold, nugget by nugget, and learn about fun statistical tools for data analysis.

Today, in the first part of our analysis, we will put our intuition about rating systems to the test. We will decode IMDb “user ratings,” those numbers such as 6.1 and 7.8 that summarize how the registered users of the IMDb rated movies on a scale from 1 to 10, typically depicted as a series of stars on the screen:

We will extract the collective wisdom of registered IMDb users in order to convert a movie’s user rating into the movie’s standing within the database. This gives us a good indicator of how the movie stacks up against other movies in general, and that’s good information to have when deciding which movies to see in the theater or add to your Netflix list.

Ready to start digging? Let’s go!

Getting to know user ratings: fundamental descriptive statistics

Like most online movie databases, the IMDb encourages its users to rate movies on a numerical scale, in this case from 1 to 10. The IMDb software averages these ratings into a composite “user rating” for each movie. King Kong, for example, currently has a user rating of 7.8. Transporter 2, on the other hand, has a user rating of 6.1.

Certainly, we have some sense of what these ratings mean. A 6.1, for example, is somewhat higher than the midpoint of the 1-to-10 scale. Thus we might expect a 6.1-rated movie to be somewhat better than the typical movie. But is this expectation justified? Also, we know a 7.8 is better than a 6.1. But how much better? Is it 1.7 stars better? And, if so, what does that mean?

To understand what user ratings mean, we must put them into context. Let’s assume that buried within the IMDb is some kind of useful information that reflects the collective wisdom of the site’s users. When a movie is rated 7.8, we will assume that the rating means the movie is “better” than lower-rated movies and “worse” than higher-rated movies. To what degree, we don’t know for sure, but that is what we are about to find out.

While we might not know what it means for a movie to be a “7.8,” we probably do have a genuine sense for what it means for a movie to be among the best of movies, or among the worst, or among the middle of the pack. We have developed this sense by experience, by watching movies over our lifetimes. What we need is some way of converting the number 7.8 into something that registers with this hard-earned experience.

As a starting point, let’s examine the most fundamental descriptive statistics of the IMDb’s user ratings:

Count	Mean	Median	St.Dev.
23,396	6.2	6.4	1.4

Breaking them down:

count – There are 23,396 user ratings in the database. (There are actually more, but to eliminate fringe movies I am considering only those movies that have been rated by more than 100 users.)
mean – The average user rating is 6.2. While some ratings are lower and others higher, if you were to put all of the ratings in a blender and purée them into a homogeneous soup, the soup’s overall rating would balance out to 6.2.
median – The rating that divides the database in half. Ratings higher than 6.4 fall into the better half; ratings lower than 6.4, the worse half.
standard deviation – This is a measure of how spread out the ratings are. Assuming the distribution of the ratings has a bell-curve shape, which we will investigate in a moment, about 85 percent of the ratings will fall within one standard deviation of the mean, i.e., in the range 6.2 +/- 1.4 = 4.8 to 7.6.

Another way to examine the ratings is graphically. The following chart, called a histogram, shows how many movies had each possible user rating:

The ratings form a pointy bell curve. It’s easy to see that few movies have ratings lower than 4 or higher than 8; most movies fall in between. The movies are most densely packed in the range that is a bit higher than 6 and a bit lower than 8. I have plotted the mean (the triangle) and median (the “X”) along the bottom of the chart to put them into perspective.

Exploring the extremes

With this information, we can begin to make crude interpretations of user ratings. Say we hear that Catwoman has a user rating of 3.4. Before we looked at the histogram, we probably could have guessed that the movie was not good. (We may even have heard as much from friends.) But now that we have seen the histogram, we know that very few movies had a rating lower than 4, let alone 3.4, and so we know the movie is among the worst ever released. It is, no pun intended, an outright dog.

On the other side of the spectrum, Batman Begins has a user rating of 8.3. Since we know that few movies rate better than 8, we know that this movie is probably among the very best.

The following histogram shows where both movies stand:

So far, we understand the extremes of the rating system. Movies lower than 4 are probably terrible, and movies higher than 8 are probably great. No doubt, that is useful information. But, what about that big lump in the middle which represents the bulk of movies? That is where there real gold is hidden. To get it, we must dig deeper.

Charting the inner masses

We already know Catwoman is bad, but how bad is it? One way to quantify its badness is to count how many movies in the database are equally bad or worse, and compare that count to the size of the entire database. In the database, there are 1,060 movies with Catwoman’s 3.4 user rating or lower. The size of the entire database is 23,396 movies. Dividing the first number by the second, we find that Catwoman is among the worst 5 percent of movies the database. It is in the 5th percentile.

We just turned a 3.4 user rating into a percentage that tells us where 3.4-rated movies stand with respect to all of the movies within the database. If we repeat the process for all possible movie ratings and plot the results, we get a chart like this:

Each point on the S-shaped curve relates a movie’s rating with its standing in the database. The circle on the lower portion of the curve, for example, represents Catwoman. Its position corresponds to a 3.4 user rating on the horizontal axis and a 0.05 portion (5 percent) on the vertical axis. Thus a 3.4-rated movie is in the 5th percentile. The triangle on the upper portion of the curve corresponds to Batman Begins, relating the movie’s 8.3 rating to its glorious standing in the 97th percentile.

Because the curve covers all ratings, not just the extremes, we now have a way to quantify the goodness or badness of middle-ground movies. Let’s return to King Kong, currently rated 7.8, and Transporter 2, currently rated 6.1. Look up their percentiles on the curve above. (Try it.) If you are careful, you should get close to the actual values of 91 and 42, respectively.

This would be a good time to reflect upon our intuition about user ratings. Earlier, we thought a 6.1 user rating suggested that a movie was somewhat better than the typical movie. Now, however, we see that a 6.1 is worth somewhat less than is typical.

Even though their ratings differ by only 1.7 user-rating units, King Kong is in the 91st percentile – very good – and Transporter 2 is way down in the 42nd percentile – not so good. To look at the difference another way, about half of the movies in the database fall in between Transporter 2 and King Kong: 0.91 – 0.42 = 0.49. A small difference in user ratings can represent a large difference in standings, which might further challenge our intuition about ratings.

Additionally, differences in standings are not proportional to differences in user ratings. Catwoman, for example, has a user rating of 3.4 and falls into the 5th percentile. Transporter 2, with its 6.1 user rating, is a whole 2.7 user-rating units away from Catwoman, but only 37 percent of movies stand between them. Even though Transporter 2 is closer to King Kong in terms of user ratings, it is really closer to Catwoman in terms of standing.

Movie-rating decoder ring

A chart is great for understanding the relationship between user ratings and movie standings, but it is not ideal for day-to-day use, when we just want to figure out where a movie stands before deciding whether it is worth watching. For times like that, a lookup table is a convenient alternative. The table below, for example, summarizes the rating-standing relationship in a convenient “decoder-ring” format. Find a movie’s rating in the left column, and the corresponding entry in the right column gives the movie’s standing.

Rating	Percentile
4.00	8
5.00	19
5.25	22
5.50	29
5.75	33
6.00	40
6.25	45
6.50	55
6.75	61
7.00	70
7.25	76
7.50	84
7.75	89
8.00	94
8.25	96
8.50	98
8.75	99
9.00	100

Using King Kong as an example again, let’s look up 7.8. It turns out that 7.8 is not in the table, but 7.75 is, and it corresponds to the 89th percentile. So we can guesstimate that King Kong is a bit above the 89th percentile, which, as we know from earlier, is correct, the actual value being 91. The decoder ring is not as precise as the chart, but it is more than good enough for finding a movie’s approximate standing quickly – something that might be handy on a Friday night.

Summary: weighing the gold

What have we dug up so far? First, we computed a few essential descriptive statistics of the IMDb’s user ratings. We learned that the average rating is 6.2 and that the median, which divides the ratings into better and worse halves, is 6.4.

Second, we plotted a histogram in order to inspect the ratings visually. Right away, we could tell that movies rated lower than 4 are among the very worst, and movies rated higher than 8 are among the very best.

Third, in order to give more meaning to ratings in between those two extremes, we turned to percentiles. We computed Catwoman’s by hand; it’s in the 5th percentile – ouch! Then we plotted a curve that represents the relationship between user ratings and percentiles. Using this curve we determined that King Kong is in the 91st percentile and Transporter 2 is in the 42nd percentile – a large difference in movie standings.

Finally, we created a tabular “decoder ring” to summarize what the curve depicted. It is a quick and easy way to find a movie’s standing given its user rating.

That concludes our first dig of the Internet Movie Database. Next time, we will examine the factors that influence movie ratings. Are Documentaries better than Horror flicks? Are old movies generally better than new movies? We will ask those questions and more in the next part of the series.

Until then, enjoy a movie or two. And don’t forget your slide-rule.

Acknowledgments

The movie information used in this article is courtesy of The Internet Movie Database and used with permission.

Second, my analysis was performed with R software from the R Project for Statistical Computing. R is a great statistics package. It’s Free Software, and it has a great community around it. Do check it out.

Open-source statistics: R and ESS

Tom Moertel — Fri, 27 Aug 2004 12:00:00 -0400

Recently, I needed to perform some statistical work. But I didn’t want use my previous tool-of-choice, Mathematica, because I decided after my switch to Linux not to rely on proprietary software when viable open-source alternatives existed. And thus I embarked on a short search for open-source statistics software.

R

My search was fruitful, leading me immediately to the delightfully GPL-licensed R Project for Statistical Computing: “R is a language and environment for statistical computing and graphics.” (The R system and language are similar to S, developed at Bell Labs.) The R language has functional-programming semantics (which I love) and supports (among others) the object-oriented style of programming, which is used extensively for R’s statistical interface. Most results in R are delivered in terms of objects, such as tables and and vectors and linear models, whose properties you can inspect and manipulate as you would expect. The underlying classes provide specialized methods for common operations so that the objects do the right things in response to generic commands.

Immediately, I was hooked on R. Despite having a sharp initial learning curve, R is straightforward to use. Once you get the lay of the land, you can reliably guess what functions and their arguments mean. The help facility is good, too, and can integrate with your web browser if you desire.

And the graphics! Graphs and charts are often the first, best way to size up data sets. R makes it easy to create publication-quality graphs and charts, drawing on any number of supported “graphical devices.” Among the stock devices are postscript, pdf, LaTeX, png, xfig, postscript-rendered bitmaps, and X11 (windows). For a tiny example of R’s graphics, see my posts on Mining gold from the Internet Movie Database.

To make the already-attractive R downright irresistible, the R community offers the Comprehensive R Archive Network (CRAN), the R equivalent of Perl’s CPAN. (One of the CRAN mirrors is hosted by Pittsburgh’s own pair networks.) CRAN provides packages for esoteric methods of analysis, database integration, genetics, time series analysis, HTTP (!), map projections, vegetation science, and myriad others. Additionally, CRAN provides numerous sample data sets, many corresponding to examples and problem sets from popular statistics textbooks. (I should note that R, out of the box, comes loaded with tools and sample data. CRAN isn’t in any way remedial but rather expands R’s initial richness to mind-blowing proportions.)

ESS

Once I started to use R frequently, I grew tired of the command-line interface. That’s where Emacs Speaks Statistics (ESS) comes in. It’s an add-on to Emacs that provides a seamless, rich interface to R (and other statistics packages). Since I live in Emacs, ESS was a natural fit for my working style. Highly recommended. (If you’re interested, I have made a Fedora/RedHat RPM package for ESS. Get it in the RPMs section of the site.)

Summary

If you’re looking for a good statistics system, get R. Now. And if you use Emacs, too, by all means get ESS. (If you just need a few bare-bones tools, however, you might want to check out my tiny statistics tools in Tom’s Perl code on the Community Projects site.)