The oddities of floating-point arithmetic

By Tom Moertel
Posted on
Tags:

Tobias at too-biased asks wtf? is going on with the following unintuitive result from Ruby:

irb(main):001:0> (4.10 * 100).to_i
=> 409

What’s going on is that toi_ truncates its floating-point argument, discarding the fractional part. The fractional part, owing to the intricacies of floating-point arithmetic, does not agree with our intuition:

irb(main):002:0> "%3.20g" % (4.10 * 100)
=> "409.99999999999994316"

In this case, the factional part is almost 1. Throwing it away turns our expected result of 410 into the unexpected 409. (That’s why round makes for a better choice than toi_ when we want the nearest integer to our float.)

If this behavior seems bizarre, I recommend David Goldberg’s classic What Every Computer Scientist Should Know About Floating Point Arithmetic (1991). The paper was originally published in ACM Computing Surveys and later republished with corrections. The corrected versions, however, were apparently typeset in Microsoft Word, and the formulas are hard to look at (especially if you are accustomed to TeX-quality math typesetting). You might want to get your hands on a PDF of the original version if you care about that kind of thing.

comments powered by Disqus