Insufficiently accurate
- Dominant language
- JavaScript
- Stars
- 199
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
The accuracy of the values goes downhill from bit 16 onward, with incorrect results.
I.e. using the 32-bit (float) toy:
1. Set only bit 17 'on': 0x3F810000 : value is 1.0078125: correct!
2. Set only bit 16 'on': 0x3F808000 : value is 1.0039062: incorrect! [Should be 1.00390625]
3. Set only bit 15 'on': 0x3F804000 : value is 1.0019531: incorrect! [Should be 1.001953125]
4. Set only bit 14 'on': 0x3F802000 : value is 1.0009766: incorrect! [Should be 1.0009765625]
The inaccuracies accumulate.
My specific egg-on-face incident was "1.28": the toy claims that 0x3FA3D70A correctly represents "1.28", but any C compiler will tell you the value is 1.27999973297.
[Or Google:
1+(2^-2)+(2^-6)+(2^-7)+(2^-8)+(2^-9)+(2^-11)+(2^-13)+(2^-14)+(2^-15)+(2^-20)]
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.