Faq/Calc/Accuracy

From The Document Foundation Wiki
< Faq‎ | Calc
Jump to: navigation, search

Template:TopMenu/en Template:Menu/en Template:Menu.Documentation/en Template:Menu.FAQ/en

Other languages:
čeština • ‎dansk • ‎English


Inherent Accuracy Problem

Calc, just like most other spreadsheet software, uses floating-point math capabilities available on hardware. Given that most contemporary hardware uses binary floating-point arithmetic with limited precision defined in IEEE 754, many decimal numbers - including as simple as 0.1 - cannot be precisely represented in Calc (which uses 64-bit double-precision numbers internally). Calculations with those numbers necessarily results in rounding errors, and those accumulate with every calculation. This is not a bug, but is expected and currently unavoidable without using complex calculations in software, which would incur inappropriate performance penalties, and thus is out of question. Users need to account for that, and use rounding and comparisons with epsilon as necessary.

An example:

A
1 31000.99
2 32000.12
3 =A1-A2

This will result in -999.129999999997 in A3, instead of expected -999.13 (you might need to increase shown decimal places in cell format to see this).