Tom Moertel's Weblog: Tag movies

How to download photos and movies from the Palm Centro to a Linux desktop

Tom Moertel — Fri, 02 Nov 2007 15:32:00 -0400

I recently got a Palm Centro smartphone, and so far I love it. Like most modern cell phones, it has a built-in camera and takes decent snapshots and even records short movies. It’s great for spur-of-the-moment shots when I don’t have my real camera. The trick – and there’s always a trick when it comes to cell phones – is getting the photos off the camera and onto my computer.

To get at my pictures, Sprint would prefer that I sign up for their ludicrously expensive “PictureMail” service. Leave it to weasely telecom execs to come up with another way to squeeze money from teenagers: charge them $5 each month for the “privilege” of sharing their pictures with friends. This fee, of course, is in addition to the fee for “unlimited” mobile Internet use. I guess picture bits are somehow more expensive to move over the air than other kinds of bits.

In any case, my next goal after getting my Centro to hotsync with my Linux workstation was to figure out how to download my photos and movies.

After a bit of hacking, I figured out that the Centro stores images in a typical digital-camera-image (DCIM) hierarchy. For example, I have a 4-GB microSD card installed in my Centro, and I store my photos in the “Palm” album on it. This album ends up stored in the /DCIM/Palm directory on the card.

Using the pilot-xfer program from the pilot-link project, I was able to find the directory and its contents. The trick was to use the sparsely documented –D flag to work with the Centro’s virtual filesystem. Here, for example, is how I list the contents of the Palm album:

$ pilot-xfer -p usb: -D /DCIM/Palm -l

   Listening for incoming connection on usb:... connected!

   Directory of /DCIM/Palm...
        652 Fri Nov  2 08:17:06 2007  Album.db
     292053 Fri Nov  2 09:04:20 2007  Photo_110207_001.jpg
      78493 Fri Nov  2 08:17:06 2007  Video_110207_001.3g2
         20 Wed Oct 31 12:09:20 2007  Thumbnail.db

   Thank you for using pilot-link.

Here, you can see that I have one photo and one movie in the album. (Movies are stored in .3g2 files that contain MPEG4 video.)

To download the files, I again turned to pilot-xfer, this time using the –f (fetch) flag to fetch a list of files. Here, for example, I’ll fetch the image from the listing above:

$ pilot-xfer -p usb: -D /DCIM/Palm -f Photo_110207_001.jpg

   Listening for incoming connection on usb:... connected!

   Fetching '/DCIM/Palm' ... (292053 bytes)   285 KiB total.

   Thank you for using pilot-link.

So that’s the process. It’s kind of clunky, so I wrote a small Python program to automate it. (I’m learning Python. If you’re a Pythonista, please consider critiquing my code. I would be especially thankful if you could point out any helpful idioms that I may have overlooked.)

Here’s how to use the program:

$ get-pilot-photos.py --help
Usage: get-pilot-photos.py [options]

Options:
  -h, --help            show this help message and exit
  -s SRCDIR, --srcdir=SRCDIR
                        VFS dir on Palm device from which to fetch images
  -d DESTDIR, --destdir=DESTDIR
                        Where to save the images on your computer

Both the —srcdir and —dstdir options are optional. If you omit the first, the program will download photos and movies from the /DCIM/Palm album. If you omit the second, the program will save the downloads to a new, timestamped directory within your home directory.

That’s it. The code is below.

#!/usr/bin/env python

# get-pilot-photos.py -
# Download photos and movies from my Palm Centro via pilot-link
#
# Tom Moertel <tom@moertel.com>
# 2007-11-01
#
# Copyright 2007 Thomas G. Moertel
#
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# See <http://www.gnu.org/licenses/> for more.

import os
import optparse
import re
import subprocess
import time

PILOT_XFER = 'pilot-xfer'
DEFAULT_PALM_IMAGE_DIR = '/DCIM/Palm'

class PhotoImporter(object):

    def __init__(self, src_dir, dest_dir=None):
        self.src_dir = src_dir
        self.dest_dir = dest_dir or self.get_image_dir()

    def run(self):
        print 'Finding images in %s on your Palm device.' % self.src_dir
        print 'Begin hotsync now...'
        images = self.get_image_list()
        if len(images) == 0:
            print 'No images were found.  Done.'
            return
        print 'Found %s images' % len(images)
        print 'Waiting for hotsync to complete...'
        time.sleep(10) # give 1st hotsync time to complete
        print 'Begin another hotsync now...'
        self.fetch_images(images)
        print 'Done.  The images were fetched to the following directory:'
        print self.dest_dir

    def get_image_list(self):
        cmdline = [PILOT_XFER, '-p',  'usb:', '-D', self.src_dir, '-l']
        proc = subprocess.Popen(cmdline, stdout=subprocess.PIPE)
        listing = proc.stdout.read()
        proc.wait()
        return re.findall(r'\b\S+\.(?:jpg|3g2)\b', listing)

    def fetch_images(self, images):
        cmdline = [PILOT_XFER, '-p',  'usb:', '-D', self.src_dir, '-f'] + images
        subprocess.Popen(cmdline, cwd=self.dest_dir).wait()

    def get_image_dir(self):
        root = os.getenv('HOME') or tempfile.mkdtemp()
        now = time.strftime("%Y-%m-%d--%H.%M.%S", time.localtime())
        dir = os.path.join(root, 'images', 'unsorted-pix', now)
        os.makedirs(dir, mode=0771)
        return dir

def main():
    p = optparse.OptionParser()
    p.add_option('--srcdir', '-s', default=DEFAULT_PALM_IMAGE_DIR,
                 help='VFS dir on Palm device from which to fetch images')
    p.add_option('--destdir', '-d',
                 help='Where to save the images on your computer')
    opts, args = p.parse_args()
    PhotoImporter(opts.srcdir, opts.destdir).run()

if __name__ == '__main__':
    main()

Netflix vs. Amazon Unbox: Netflix still wins

Tom Moertel — Sat, 07 Apr 2007 12:20:00 -0400

When Amazon.com announced its its Unbox video-download service, I was skeptical. Compared to the reigning champion – the DVD – Unbox looked like a loser:

Unbox burdened its customers with DRM and the annoyances that come with DRM
Unbox required the use of a Windows-only player application
Unbox movies lacked “standard” DVD features such as surround sound, alternative audio tracks, commentaries, and bloopers

The first two points were deal-breakers, so I wrote off Unbox and did my best to ignore it.

And then Amazon hooked up with TiVo. Beaming movies directly into my TiVo box eliminates the need to deal with DRM and Windows annoyances. My two big concerns sidestepped, I decided to give Unbox another look. I still wouldn’t want to buy Unbox-to-TiVo movies because they lack the typical DVD extras and would tie up storage space on my TiVo, but Unbox might be a decent way to rent the occasional movie – if the price were right.

Is the price right?

That depends on how the price of Unbox compares with the price of my current rental option of choice, Netflix. Both services offer immediate access to good movies: Unbox by on-demand downloads, Netflix by ensuring that I almost always have a DVD or two in the house.

To compare Unbox with Netflix, I had to figure out how much a rental costs me with each service. With Unbox the figuring was easy because each rental has its own price tag, typically $3.99.

With Netflix, it’s a bit trickier because the rental price depends upon how many DVDs I rent in a month. I pay a monthly fee of $17.99 and can rent as many DVDs as I want, at least until the infamous Netflix rate throttle kicks in. To determine how many DVDs I rent during the typical month, I had to download my rental history. (If you’re a Netflix subscriber, you can get your history from the Returned Rentals page.) After downloading my history, massaging it into the desired form, and loading it into R, I generated a stem-and-leaf plot to visualize the number of DVDs I have rented during each of the 76 months I have been a Netflix subscriber:

> stem(monthly.rental.counts, scale=2)

  The decimal point is at the |

   1 | 0
   2 | 000
   3 | 0000000
   4 | 00000000000
   5 | 000000000000
   6 | 000000000000000
   7 | 0000
   8 | 000000
   9 | 00000
  10 | 0000
  11 | 0
  12 | 00
  13 | 00
  14 | 00
  15 | 0

It looks like I have rented as few as one and as many as fifteen DVDs in a month. Most months, however, I rent between three and ten DVDs. On average, I rent about 6.4 DVDs per month:

> summary(monthly.rental.counts)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.000   4.000   6.000   6.408   8.000  15.000

Thus my average rental price is about $2.80 per DVD:

> 17.99 / 6.4
[1] 2.810937

Now I can make my Unbox-vs-Netflix price comparison. For me, it looks like Unbox is about 40 percent more expensive than Netflix:

> 3.99 / 2.81
[1] 1.419929

So the price of Unbox is not right, at least for me.

Testing Unbox-to-TiVo rentals

Because Amazon is offering free $15 credits to TiVo owners, I decided to give Unbox a test drive. My test rental was The Illusionist. Renting the movie was easy (just one click), and shortly thereafter Unbox automatically downloaded the movie to my TiVo box. When I played the movie, however, I was disappointed with the video quality. I easily noticed banding artifacts, which were distracting at times. On the whole, the viewing experience was inferior to watching a DVD.

Netflix still beats Unbox

For me, then, Unbox is still a loser. It costs more and delivers less than DVD rentals via Netflix.

A note to my friends at Amazon.com

I would be happy to give you my business, but right now you’re not earning it. If you want me as an Unbox customer, here is the recipe for winning me over:

Let me easily download movie rentals to my TiVo. (Check.)
Offer true DVD quality or better. (You’re not there yet.)
Sell the rentals for less than $2.80. (You’re not there yet.)

Until then, I’ll have to give my money to Netflix.

Cheers,
Tom

Update: edits for clarity; added tags.

The IMDB Movie Rating Decoder Ring: updated w/ 2 March 2007 data

Tom Moertel — Fri, 09 Mar 2007 17:40:00 -0500

If you want to get more out of IMDB movie ratings, check out my IMDB Movie Rating Decoder Ring, now updated with fresher data (as of 2 March 2007).

Mining gold from the Internet Movie Database, part 1: decoding user ratings

Tom Moertel — Tue, 17 Jan 2006 20:59:00 -0500

The Internet Movie Database (IMDb) is a rich source of online movie information. The problem is, the true gold is buried deep beneath the site’s user-friendly exterior and hidden within the database itself. With a little digging, however, we can extract the gold, nugget by nugget, and learn about fun statistical tools for data analysis.

Today, in the first part of our analysis, we will put our intuition about rating systems to the test. We will decode IMDb “user ratings,” those numbers such as 6.1 and 7.8 that summarize how the registered users of the IMDb rated movies on a scale from 1 to 10, typically depicted as a series of stars on the screen:

We will extract the collective wisdom of registered IMDb users in order to convert a movie’s user rating into the movie’s standing within the database. This gives us a good indicator of how the movie stacks up against other movies in general, and that’s good information to have when deciding which movies to see in the theater or add to your Netflix list.

Ready to start digging? Let’s go!

Getting to know user ratings: fundamental descriptive statistics

Like most online movie databases, the IMDb encourages its users to rate movies on a numerical scale, in this case from 1 to 10. The IMDb software averages these ratings into a composite “user rating” for each movie. King Kong, for example, currently has a user rating of 7.8. Transporter 2, on the other hand, has a user rating of 6.1.

Certainly, we have some sense of what these ratings mean. A 6.1, for example, is somewhat higher than the midpoint of the 1-to-10 scale. Thus we might expect a 6.1-rated movie to be somewhat better than the typical movie. But is this expectation justified? Also, we know a 7.8 is better than a 6.1. But how much better? Is it 1.7 stars better? And, if so, what does that mean?

To understand what user ratings mean, we must put them into context. Let’s assume that buried within the IMDb is some kind of useful information that reflects the collective wisdom of the site’s users. When a movie is rated 7.8, we will assume that the rating means the movie is “better” than lower-rated movies and “worse” than higher-rated movies. To what degree, we don’t know for sure, but that is what we are about to find out.

While we might not know what it means for a movie to be a “7.8,” we probably do have a genuine sense for what it means for a movie to be among the best of movies, or among the worst, or among the middle of the pack. We have developed this sense by experience, by watching movies over our lifetimes. What we need is some way of converting the number 7.8 into something that registers with this hard-earned experience.

As a starting point, let’s examine the most fundamental descriptive statistics of the IMDb’s user ratings:

Count	Mean	Median	St.Dev.
23,396	6.2	6.4	1.4

Breaking them down:

count – There are 23,396 user ratings in the database. (There are actually more, but to eliminate fringe movies I am considering only those movies that have been rated by more than 100 users.)
mean – The average user rating is 6.2. While some ratings are lower and others higher, if you were to put all of the ratings in a blender and purée them into a homogeneous soup, the soup’s overall rating would balance out to 6.2.
median – The rating that divides the database in half. Ratings higher than 6.4 fall into the better half; ratings lower than 6.4, the worse half.
standard deviation – This is a measure of how spread out the ratings are. Assuming the distribution of the ratings has a bell-curve shape, which we will investigate in a moment, about 85 percent of the ratings will fall within one standard deviation of the mean, i.e., in the range 6.2 +/- 1.4 = 4.8 to 7.6.

Another way to examine the ratings is graphically. The following chart, called a histogram, shows how many movies had each possible user rating:

The ratings form a pointy bell curve. It’s easy to see that few movies have ratings lower than 4 or higher than 8; most movies fall in between. The movies are most densely packed in the range that is a bit higher than 6 and a bit lower than 8. I have plotted the mean (the triangle) and median (the “X”) along the bottom of the chart to put them into perspective.

Exploring the extremes

With this information, we can begin to make crude interpretations of user ratings. Say we hear that Catwoman has a user rating of 3.4. Before we looked at the histogram, we probably could have guessed that the movie was not good. (We may even have heard as much from friends.) But now that we have seen the histogram, we know that very few movies had a rating lower than 4, let alone 3.4, and so we know the movie is among the worst ever released. It is, no pun intended, an outright dog.

On the other side of the spectrum, Batman Begins has a user rating of 8.3. Since we know that few movies rate better than 8, we know that this movie is probably among the very best.

The following histogram shows where both movies stand:

So far, we understand the extremes of the rating system. Movies lower than 4 are probably terrible, and movies higher than 8 are probably great. No doubt, that is useful information. But, what about that big lump in the middle which represents the bulk of movies? That is where there real gold is hidden. To get it, we must dig deeper.

Charting the inner masses

We already know Catwoman is bad, but how bad is it? One way to quantify its badness is to count how many movies in the database are equally bad or worse, and compare that count to the size of the entire database. In the database, there are 1,060 movies with Catwoman’s 3.4 user rating or lower. The size of the entire database is 23,396 movies. Dividing the first number by the second, we find that Catwoman is among the worst 5 percent of movies the database. It is in the 5th percentile.

We just turned a 3.4 user rating into a percentage that tells us where 3.4-rated movies stand with respect to all of the movies within the database. If we repeat the process for all possible movie ratings and plot the results, we get a chart like this:

Each point on the S-shaped curve relates a movie’s rating with its standing in the database. The circle on the lower portion of the curve, for example, represents Catwoman. Its position corresponds to a 3.4 user rating on the horizontal axis and a 0.05 portion (5 percent) on the vertical axis. Thus a 3.4-rated movie is in the 5th percentile. The triangle on the upper portion of the curve corresponds to Batman Begins, relating the movie’s 8.3 rating to its glorious standing in the 97th percentile.

Because the curve covers all ratings, not just the extremes, we now have a way to quantify the goodness or badness of middle-ground movies. Let’s return to King Kong, currently rated 7.8, and Transporter 2, currently rated 6.1. Look up their percentiles on the curve above. (Try it.) If you are careful, you should get close to the actual values of 91 and 42, respectively.

This would be a good time to reflect upon our intuition about user ratings. Earlier, we thought a 6.1 user rating suggested that a movie was somewhat better than the typical movie. Now, however, we see that a 6.1 is worth somewhat less than is typical.

Even though their ratings differ by only 1.7 user-rating units, King Kong is in the 91st percentile – very good – and Transporter 2 is way down in the 42nd percentile – not so good. To look at the difference another way, about half of the movies in the database fall in between Transporter 2 and King Kong: 0.91 – 0.42 = 0.49. A small difference in user ratings can represent a large difference in standings, which might further challenge our intuition about ratings.

Additionally, differences in standings are not proportional to differences in user ratings. Catwoman, for example, has a user rating of 3.4 and falls into the 5th percentile. Transporter 2, with its 6.1 user rating, is a whole 2.7 user-rating units away from Catwoman, but only 37 percent of movies stand between them. Even though Transporter 2 is closer to King Kong in terms of user ratings, it is really closer to Catwoman in terms of standing.

Movie-rating decoder ring

A chart is great for understanding the relationship between user ratings and movie standings, but it is not ideal for day-to-day use, when we just want to figure out where a movie stands before deciding whether it is worth watching. For times like that, a lookup table is a convenient alternative. The table below, for example, summarizes the rating-standing relationship in a convenient “decoder-ring” format. Find a movie’s rating in the left column, and the corresponding entry in the right column gives the movie’s standing.

Rating	Percentile
4.00	8
5.00	19
5.25	22
5.50	29
5.75	33
6.00	40
6.25	45
6.50	55
6.75	61
7.00	70
7.25	76
7.50	84
7.75	89
8.00	94
8.25	96
8.50	98
8.75	99
9.00	100

Using King Kong as an example again, let’s look up 7.8. It turns out that 7.8 is not in the table, but 7.75 is, and it corresponds to the 89th percentile. So we can guesstimate that King Kong is a bit above the 89th percentile, which, as we know from earlier, is correct, the actual value being 91. The decoder ring is not as precise as the chart, but it is more than good enough for finding a movie’s approximate standing quickly – something that might be handy on a Friday night.

Summary: weighing the gold

What have we dug up so far? First, we computed a few essential descriptive statistics of the IMDb’s user ratings. We learned that the average rating is 6.2 and that the median, which divides the ratings into better and worse halves, is 6.4.

Second, we plotted a histogram in order to inspect the ratings visually. Right away, we could tell that movies rated lower than 4 are among the very worst, and movies rated higher than 8 are among the very best.

Third, in order to give more meaning to ratings in between those two extremes, we turned to percentiles. We computed Catwoman’s by hand; it’s in the 5th percentile – ouch! Then we plotted a curve that represents the relationship between user ratings and percentiles. Using this curve we determined that King Kong is in the 91st percentile and Transporter 2 is in the 42nd percentile – a large difference in movie standings.

Finally, we created a tabular “decoder ring” to summarize what the curve depicted. It is a quick and easy way to find a movie’s standing given its user rating.

That concludes our first dig of the Internet Movie Database. Next time, we will examine the factors that influence movie ratings. Are Documentaries better than Horror flicks? Are old movies generally better than new movies? We will ask those questions and more in the next part of the series.

Until then, enjoy a movie or two. And don’t forget your slide-rule.

Acknowledgments

The movie information used in this article is courtesy of The Internet Movie Database and used with permission.

Second, my analysis was performed with R software from the R Project for Statistical Computing. R is a great statistics package. It’s Free Software, and it has a great community around it. Do check it out.