WebAssembly / WebAssembly/spec
Double rounding fix is (still) wrong in Windows
Nobody has claimed this yet.
- Dominant language
- WebAssembly
- Stars
- 3.5k
- Forks
- 539
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 13
Description
Found during the development of WAH with a binary script converted from const.wast. Minimal reproduction:
(module (func (export "f") (result f32) (f32.const +8.8817847263968443574e-16)))
(assert_return (invoke "f") (f32.const +0x1.000002p-50))
(Note that a similar construction in hexadecimal literals, (f32.const +0x1.00000100000000001p-50), correctly rounds to 0x1.000002p-50.)
The value in question is almost in the middle of two consecutive binary32 fp number 0x1.0p-50 and 0x1.000002p-50, but it is not the exact middle and in fact very slightly closer to 0x1.000002p-50:
d=8.88178419700125232338905334472656250000000000e-16
+0.00000052939559203401094665527343750000000000e-16
----------------------------------------------------
n=8.88178472639684435740000000000000000000000000e-16
+0.000000529395592033864477180129688349552452..e-16
----------------------------------------------------
u=8.881785255792436391264477180129688349552452..e-16
The root cause is the inaccuracy of fxx.ml line 285. String.length s here is 25 (20 significant digits + . + e-16), so the format string is %.25g, but %g is platform-dependent! (source) In my experience, msvcrt's printf notably does not give any more significant digits than the type's own limit (even %.100f gives ~18 significant digits followed by zeroes), so %.25g actually doesn't make any difference with msvcrt. The correct approach would be to use an actual bigint (scaled by, say, 5^1100 to elide fractional digits).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at fxx.ml line 285 and reproduce the minimal f32.const case on Windows, comparing it with the hexadecimal-literal case. Trace how the platform-dependent %.25g formatting affects conversion and verify that the decimal value rounds to 0x1.000002p-50 without relying on msvcrt precision limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ocaml, wasm
- Domain
- compilers, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100