Posted by Tom Moertel
Sun, 26 Aug 2007 01:56:00 GMT
The R statistics system can produce
first-class data visualizations, commonly known as plots. Internally,
plots are represented in an abstract graphics format that can be
rendered on any of R’s wide range of graphics “devices” to produce
concrete output – windows, bitmap files, PostScript files, PDF files,
and others.
The bitmap formats, such as PNG, are preferred for posting
plots online because of their widespread support by web browsers. The
default bitmap-rendering devices in R, unfortunately, produce graphics
that look a little too “bitmapped” for modern web tastes. Here, for example,
is a plot rendered by R’s “png” device:
There’s nothing technically wrong with the plot, but it looks out
of place on a web page. That’s because modern web
browsers use font-smoothing and anti-aliasing techniques to render
just about everything else on the page. Against this clean, un-jagged
backdrop, the oh-so-bitmapped plot looks like a throwback to
a previous era.
Happily, we can produce clean, anti-aliased R plots with a little
help. Here’s the earlier plot, anti-aliased:
To produce the anti-aliased plot, I used R to produce a PDF file. Then I
rendered the PDF file into a PNG image at 300 dpi using Ghostscript.
Finally, I scaled the 300-dpi image down to screen resolution,
producing a high-quality, anti-aliased result.
Here’s the recipe in detail.
First, I define an R function called pdfit that takes an
abstract graphics object and makes a PDF-file rendering of it, using
my preferred graphics-device settings:
require("lattice")
pdfit <- function(f, ...) {
trellis.device(dev=pdf, theme="col.whitebg", ...);
print(f);
dev.off()
}
Then, when I create a plot I want to publish, I use pdfit to render
it into a PDF file:
P.img <- xyplot( subs.low + subs.high ~ date, ... )
pdfit(P.img, file="image-downloads.pdf") # render plot into PDF file
Finally, I use Ghostscript and
ImageMagick to convert the PDF file into
a high-quality, anti-aliased PNG file. (I keep both formats: the PDF
file is best for publishing in printed papers, and the PNG file is
best for posting online.) I use a simple Makefile to automate the
process of converting the PDF files into PNG files:
# Makefile (GNU make)
pdfs := $(wildcard *.pdf)
pngs := $(pdfs:.pdf=.png)
all: $(pngs)
.PHONY: all
%.png: %.pdf
gs -dSAFTER -dBATCH -dNOPAUSE -sDEVICE=png16m \
-dGraphicsAlphaBits=4 -dTextAlphaBits=4 -r300 \
-dBackgroundColor='16#ffffff' \
-sOutputFile=$@ > /dev/null \
$< && \
mogrify -resize 500 $@
With this Makefile in my graphics directory, just a single “make”
command is all it takes to convert my PDF images into
anti-aliased PNG files, ready to post online.
And that’s it.
Do you have any tips or tricks for making good-looking graphics with
R? If so, please do share.
Update: There is one downside to the sexy, anti-aliased plots: they
are not as compressible as the old-style jagged plots. For the
images above, for instance, the anti-aliased PNG file weighs in
at 45 KB, but the original PNG file is a feathery 4.7 KB.
So, if bandwidth is precious to you – or you’re planning on getting
Slashdotted – you might want to stick
with the jaggies.
Posted in statistics
Tags graphics, plots, R, statistics, tips, tricks
8 comments
no trackbacks
Posted by Tom Moertel
Thu, 23 Aug 2007 01:34:00 GMT
As everybody knows, statistics is fun. Is there
anything cooler than crushing a heap of seemingly uninteresting
numbers into gleaming jewels of meaning? Of course not! Models,
data-visualization plots, and fat data sets are way cool.
So, let’s find an excuse to play with them.
Here’s an excuse –
I mean, an important and highly relevant question that many of us share:
How many people actually read our blogs? To answer the
question, we will need to use statistics, data, and cool plots.
Further, if you’ve got the raw data for your blog, you can follow
along with your own analysis. Even more fun!
We’ll start with a simple inspection of common web-log data, using
command-line tools. After developing a rough understanding of what
useful information we can extract, we’ll analyze the raw data using a
series of successively more sophisticated techniques. In the end, we
will derive a simple formula for estimating readership from easily
obtainable data.
Sound good? Then let’s get rocking.
But first, a preemptive strike on would-be poo-pooers: I know all about
FeedBurner. I know they will track my blog’s subscribers and use
their mystical powers to infer the number of “real” subscribers I
have. I know it’s all so easy. But easy isn’t the point. I want to
understand what’s going on. Just taking somebody’s word for it isn’t
nearly as satisfying as figuring it out yourself – nor as fun.
OK. For real this time, let’s get rocking.
Read more...
Posted in statistics
Tags blog, fun, modeling, R, statistics
5 comments
no trackbacks
Posted by Tom Moertel
Wed, 25 Apr 2007 18:07:00 GMT
Just a quick note for folks using the R statistics
system on Fedora
Linux. I have packaged for Fedora a
bunch of R packages from the CRAN. (R
packages have to be packaged again, as RPM packages, to integrate with
Fedora Linux.)
My initial goal was to package
arm,
which contains tools for working with various regression models.
(This package accompanies Andrew Gelman and Jennifer Hill’s wonderful
book Data Analysis Using Regression and Multilevel/Hierarchical Models.)
Packaging “arm,” however, quickly snowballed into packaging a bunch of
prerequisites. Thankfully, I have now completed that task and can
share the fruits of my labor with you.
All in all, to install “arm,” you will need the following RPMs:
- R-arm-1.0-2
- R-car-1.2-1
- R-lme4-0.9975-1
- R-Matrix-0.9975-1
- R-R2WinBUGS-2.0-1
The following RPMs are optional (but you will need them if you
want to rebuild the RPMs):
- R-coda-0.10-1
- R-leaps-2.7-1
- R-mlmRev-0.995-1
You can download the packages from the RPMs
section of the Community
Projects site. Better yet, you can use Yum to
download them for you. Just add the moertel-community
Yum repository to your /etc/yum.repos.d directory (see RPMs for the recipe) and then use the
following command:
$ sudo yum install R-arm
Yum will automatically resolve dependencies and install the required
packages. If you want any of the optional packages, add them after
“R-arm” on the command line.
I have built the packages for Fedora Core 6 on the x86_64 architecture, but the
RPM specs are available
if you want to rebuild the packages for other architectures. (See
the instructions for rebuilding RPMs for help.)
Caveat:
I’m not sure that the R-R2WinBUGS package is fully functional. It
depends on BRugs, which doesn’t yet build on the Linux platform. To
get around this problem, I made R-R2WinBUGS’s dependency on BRugs
weak; the first package no longer requires the second to install.
Posted in statistics, linux
Tags fedora, R, rpms, statistics
no comments
no trackbacks
Posted by Tom Moertel
Tue, 17 Apr 2007 07:45:00 GMT
Today I wanted to extract the data that were visualized in a
chart I saw on Seth Roberts’s blog. That is, I had a picture of a data set, and I wanted the numbers behind the picture.
This task turned out to be surprisingly easy – once I found Engauge Digitizer, an open-source (GPL) tool made for this very task. After I launched Engauge, the digitization process was straightforward:
- I established the chart’s coordinate system by clicking in the corners and entering the associated coordinates.
- Then I had Engauge identify data points. With the mouse, I selected a data point by hand, teaching Engauge what a point looks like. Then Engauge identified spots on chart that looked like data points and locked on to them. I was able to step through the points to tell Engauge to skip the few it misidentified.
- I manually selected a few more data points that were scrunched into blobs and had eluded Engauge’s point-detection heuristics.
- Finally, I exported the data set in CSV format.
If you ever need to extract the data behind a chart, do check out
Engauge Digitizer. (If you use Fedora Linux,
you’ll be happy to know that I have packaged Engauge for you.
Get it at the RPMs section
of the community site.)
Posted in statistics
Tags charts, data, fedora, plots, rpms, statistics, tools
no comments
no trackbacks
Posted by Tom Moertel
Fri, 09 Mar 2007 22:40:00 GMT
If you want to get more out of IMDB movie ratings, check out my
IMDB Movie Rating Decoder Ring, now updated with fresher data (as of 2 March 2007).
Posted in statistics
Tags data, decoder_rinng, imdb, movies, ratings, stars, statistics
1 comment
no trackbacks
Posted by Tom Moertel
Thu, 20 Jul 2006 14:43:00 GMT
Every so often I check the stats for my blog to see which articles are
popular and learn how people are finding the blog. I just checked the
stats today, and, if you are curious, here is a summary of what I found.
Most popular articles
As of today, for the month of July 2006, the ten most-visited articles on my blog are as follows:
The numbers in parentheses are the articles’ relative-popularity
scores. Each score tells you how popular an article was when compared
to the blog’s root page, which has a reference score of 1000.
Most popular search keywords
The ten most popular search keywords that led people to my blog are
as follows:
Many of these were combined with other terms as part of a search phrase.
For example, “rails” was often used in the following phrases:
Spam
Over the last 30 days, there have been on average 244 attempts per day to post
spam to my blog (stdev = 96).
That’s it
And that’s the stats!
Posted in site news, statistics
no comments
no trackbacks
Posted by Tom Moertel
Fri, 20 Jan 2006 23:02:00 GMT
Every so often, I am going to write about
wondrous oddities – obscure programming-language features
that are so cool they deserve wider notice.
Today, in the first installment, I want to show you the function-call
semantics of R, a great system
for statistical computing.
You might not expect a statistics system to have a first-class
programming language at it’s heart, but if you think about it, it does
make sense. The R language, actually a dialect of the S language, is
described as “a well-developed, simple and effective programming
language which includes conditionals, loops, user-defined recursive
functions and input and output facilities.” All true. It gives me
the feeling of an infix Lisp or Scheme whose syntax is slanted toward
mathematics and vector operations. The language has an object layer,
too, but that’s not why we are here.
No, we are here to look at R’s uncommonly interesting function-call semantics, in particular
argument binding and evaluation. Let’s dig in.
Read more...
Posted in programming languages, statistics, wondrous oddities
Tags languages, R
2 comments
no trackbacks
Posted by Tom Moertel
Wed, 18 Jan 2006 01:59:00 GMT
The Internet Movie Database (IMDb) is a rich source
of online movie information. The problem is, the true gold is buried
deep beneath the site’s user-friendly exterior and hidden within the
database itself. With a little digging, however, we can extract the
gold, nugget by nugget, and learn about fun statistical tools for data
analysis.
Today, in the first part of our analysis, we will put our intuition
about rating systems to the test. We will decode IMDb “user ratings,”
those numbers such as 6.1 and 7.8 that summarize how the registered
users of the IMDb rated movies on a scale from 1 to 10, typically
depicted as a series of stars on the screen:
We will extract the collective wisdom of registered IMDb users in
order to convert a movie’s user rating into the movie’s standing
within the database. This gives us a good indicator of how the movie
stacks up against other movies in general, and that’s good information
to have when deciding which movies to see in the theater or add to
your Netflix list.
Ready to start digging? Let’s go!
Read more...
Posted in movies, statistics
Tags imdb, movies, R, statistics
5 comments
no trackbacks
Posted by Tom Moertel
Mon, 12 Sep 2005 18:12:00 GMT
I just posted new RPMs for Emacs Speaks Statistics. These RPMs are for the recent ESS 5.2.10 release. As usual, you can get them from the RPMs section of the Community Projects site.
I had to tweak the RPM spec file to accommodate build-system changes introduced in ESS 5.2.10. If you want to build from scratch, make sure you pull the most recent spec file from my site instead of using the one included in the 5.2.10 distribution, which is now out of date.
Posted in rpms, statistics
no comments
no trackbacks
Posted by Tom Moertel
Tue, 02 Nov 2004 17:00:00 GMT
The Presidential-race polling data on www.electoral-vote.com are great, and the site’s plots do help visualize the country-wide race. But I want to see all the historical data for all of the states at a glance. To this end, I have taken the data from the site (as of the morning of November 2, 2004) and created this all-inclusive summary plot that shows the state-by-state polling results from September 1, 2004 through November 2, 2004:
Some things to observe:
- Battleground states are polled frequently; others, hardly at all
- Wisconsin’s polling data suggest that Bush once had the state but lost it
See anything else interesting?
If you want to see the code I used to generate the plot, read on.
Read more...
Posted in statistics, interesting stuff
no comments
no trackbacks
readers
