evanw / evanw/float-toy

Insufficiently accurate

Open
#9 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
199
Forks
53
PR merge metrics
No merged PRs in 30d

Description

The accuracy of the values goes downhill from bit 16 onward, with incorrect results.

I.e. using the 32-bit (float) toy:
1. Set only bit 17 'on': 0x3F810000 : value is 1.0078125: correct!
2. Set only bit 16 'on': 0x3F808000 : value is 1.0039062: incorrect! [Should be 1.00390625]
3. Set only bit 15 'on': 0x3F804000 : value is 1.0019531: incorrect! [Should be 1.001953125]
4. Set only bit 14 'on': 0x3F802000 : value is 1.0009766: incorrect! [Should be 1.0009765625]

The inaccuracies accumulate.

My specific egg-on-face incident was "1.28": the toy claims that 0x3FA3D70A correctly represents "1.28", but any C compiler will tell you the value is 1.27999973297.

[Or Google:
1+(2^-2)+(2^-6)+(2^-7)+(2^-8)+(2^-9)+(2^-11)+(2^-13)+(2^-14)+(2^-15)+(2^-20)]

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.