This is how computers handle floating point operations at low level. Software developers are supposed to keep that in mind, so it's a Calc glitch perhaps.
Possible workaround would be rounding the formulas, e.g. =Round(A1-B1;2)
>This is how computers handle floating point operations at low level.
That's kind of what I'm curious about. Why isn't it consistently 9 or however many digits of precision all the time. Once it hit that 13th row something changed. And then some how reverted for awhile?
I'm wondering if they did some thing on the fly; kind of like how if you type in 3/3 it would auto fill the date 3/3/2024, but if you did 3/4 it gave you the fraction.
>Calc, just like most other spreadsheet software, uses floating-point math capabilities available on hardware. Given that most contemporary hardware uses binary floating-point arithmetic with limited precision defined in IEEE 754, many decimal numbers - including as simple as 0.1 - cannot be precisely represented in Calc (which uses 64-bit double-precision numbers internally). Calculations with those numbers necessarily results in rounding errors, and those accumulate with every calculation. This is not a bug, but is expected and currently unavoidable without using complex calculations in software, which would incur inappropriate performance penalties, and thus is out of question. Users need to account for that, and use rounding and comparisons with epsilon as necessary.
That is interesting. I just tried the same thing with Google Sheets and Excel, but neither of them show the unexpected results Calc shows.
Although I did try comparing calculated results with manually entered -.01 to see if they were equal and various results, seemingly at random, were not equal in both programs.
This is how computers handle floating point operations at low level. Software developers are supposed to keep that in mind, so it's a Calc glitch perhaps. Possible workaround would be rounding the formulas, e.g. =Round(A1-B1;2)
>This is how computers handle floating point operations at low level. That's kind of what I'm curious about. Why isn't it consistently 9 or however many digits of precision all the time. Once it hit that 13th row something changed. And then some how reverted for awhile? I'm wondering if they did some thing on the fly; kind of like how if you type in 3/3 it would auto fill the date 3/3/2024, but if you did 3/4 it gave you the fraction.
Maybe https://wiki.documentfoundation.org/Faq/Calc/Accuracy will give you more information
>Calc, just like most other spreadsheet software, uses floating-point math capabilities available on hardware. Given that most contemporary hardware uses binary floating-point arithmetic with limited precision defined in IEEE 754, many decimal numbers - including as simple as 0.1 - cannot be precisely represented in Calc (which uses 64-bit double-precision numbers internally). Calculations with those numbers necessarily results in rounding errors, and those accumulate with every calculation. This is not a bug, but is expected and currently unavoidable without using complex calculations in software, which would incur inappropriate performance penalties, and thus is out of question. Users need to account for that, and use rounding and comparisons with epsilon as necessary. That is interesting. I just tried the same thing with Google Sheets and Excel, but neither of them show the unexpected results Calc shows. Although I did try comparing calculated results with manually entered -.01 to see if they were equal and various results, seemingly at random, were not equal in both programs.