The oddities of floating-point arithmetic

Tom Moertel — Tue, 27 Sep 2005 22:59:00 -0400

Tobias at too-biased asks wtf? is going on with the following unintuitive result from Ruby:

irb(main):001:0> (4.10 * 100).to_i
=> 409

What’s going on is that to_i truncates its floating-point argument, discarding the fractional part. The fractional part, owing to the intricacies of floating-point arithmetic, does not agree with our intuition:

irb(main):002:0> "%3.20g" % (4.10 * 100)
=> "409.99999999999994316"

In this case, the factional part is almost 1. Throwing it away turns our expected result of 410 into the unexpected 409. (That’s why round makes for a better choice than to_i when we want the nearest integer to our float.)

If this behavior seems bizarre, I recommend David Goldberg’s classic What Every Computer Scientist Should Know About Floating Point Arithmetic (1991). The paper was originally published in ACM Computing Surveys and later republished with corrections. The corrected versions, however, were apparently typeset in Microsoft Word, and the formulas are hard to look at (especially if you are accustomed to TeX-quality math typesetting). You might want to get your hands on a PDF of the original version if you care about that kind of thing.

Tom Moertel's Weblog: The oddities of floating-point arithmetic

The oddities of floating-point arithmetic