Humpty Promotes > Rfo-Basic > Project Articles >

To Round or not to round ?  (Aug 2015)

Along my bouts of hobbyist programming, I often forget that computers are essentially binary devices.
i.e They are reliant on base 2.

Consider the following basic program.

print a

Now look what happens if we do this.

print a/b

Human beings are primarily base 10 creatures. By intuition, we 'know' that the result should be exactly 0.23.

At first glance, this particular basic interpreter seems to print the result without rounding.
The truth in fact is the opposite. But there is also something else going on.

To explain this, let's see how this result is stored.

Most computers store real numbers as 64-bit IEEE 754 binary format also known as 'binary64'.

Storage is Finite.

In binary, some numbers cannot be exactly represented, much the same way as 1/3 cannot be exactly represented in decimal base 10 floating point (0.3333..)
In addition, there will be other numbers that also need more digits than is available to accurately represent it's value.

So these numbers have to be approximated with either one of the neighboring (exact) numbers on it's number line. i.e it is rounded up or down to a value slightly greater or less than it's true value.

In the example, a candidate that fits this description (for binary), is variable assignment  a=2.3

2.3 in binary64 representation is
2.2999999999999998 (17 significant digits)

This value is how 2.3 is stored internally as double-precision floating point binary. It is the closest value that 64-bit IEEE 754 base 2 can get to 2.3 given the limited number of bits. It has also been rounded after being truncated to the last digit that fits in it's storage size.

When printing back this number, this version of basic happens to use java, which in-turn uses Double.toString, which will try to find the least number of digits that can 'represent' the same value.

Effectively, it rounds back up and finds the representative "2.3".
So if you "print a" you will see "2.3".

The storage hasn't changed value (still 2.2999999999999998), but the printed output has.

The same is not true for a/b.
After the arithmetic calculation of a/b, of which a is already internally inexact,
the result amplifies the precision error, which when printed, does not find "0.23" to be a representative,
and ends up printing the full 17 (significant) digit decimal "0.22999999999999998".

Such results happen for other types of arithmetic, even simple ones;
print a+b

Again, either one or both operands, stored as inexact values have been involved in a calculation which has emphasized the precision error to something unrecoverable by the print process.

This is just one simple calculation, imagine what happens if you are crunching many numbers in a loop ?


For both examples, a solution is to just Round the result to produce a 'correct' answer.

But what if we don't know anything about a and b ? i.e Perhaps there
will be an a and b that 'should' produce 0.22999999999999998 or 11.399999999999999 ?

In Java there are types like BigDecimal which give control over representation and rounding.
Another way is to convert to integer, align precision, calculate, and then convert back to real.
These types of solutions could prove expensive or cumbersome.

Like it or not, floating point numbers in devices today use base 2 storage.
What you see may be what you get, but might not be what is there.
Arithmetic can amplify errors.
And you might end up counting on your fingers after all.


Sources and Acknowledgements:

Support my projects!

Leading Cloud Surveillance service

Leading Enterprise Cloud IT Service Since 2003

Powered by, a division of DriveHQ, the leading Cloud IT and Cloud Surveillance Service provider since 2003.