WebAssembly / WebAssembly/spec

Double rounding fix is (still) wrong in Windows

Open
#2,150 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
WebAssembly
Stars
3.5k
Forks
539
Avg merge
2d 1h
Merged PRs (30d)
13

Description

Found during the development of WAH with a binary script converted from const.wast. Minimal reproduction:

(module (func (export "f") (result f32) (f32.const +8.8817847263968443574e-16)))
(assert_return (invoke "f") (f32.const +0x1.000002p-50))

(Note that a similar construction in hexadecimal literals, (f32.const +0x1.00000100000000001p-50), correctly rounds to 0x1.000002p-50.)

The value in question is almost in the middle of two consecutive binary32 fp number 0x1.0p-50 and 0x1.000002p-50, but it is not the exact middle and in fact very slightly closer to 0x1.000002p-50:

d=8.88178419700125232338905334472656250000000000e-16
 +0.00000052939559203401094665527343750000000000e-16
----------------------------------------------------
n=8.88178472639684435740000000000000000000000000e-16
 +0.000000529395592033864477180129688349552452..e-16
----------------------------------------------------
u=8.881785255792436391264477180129688349552452..e-16

The root cause is the inaccuracy of fxx.ml line 285. String.length s here is 25 (20 significant digits + . + e-16), so the format string is %.25g, but %g is platform-dependent! (source) In my experience, msvcrt's printf notably does not give any more significant digits than the type's own limit (even %.100f gives ~18 significant digits followed by zeroes), so %.25g actually doesn't make any difference with msvcrt. The correct approach would be to use an actual bigint (scaled by, say, 5^1100 to elide fractional digits).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at fxx.ml line 285 and reproduce the minimal f32.const case on Windows, comparing it with the hexadecimal-literal case. Trace how the platform-dependent %.25g formatting affects conversion and verify that the decimal value rounds to 0x1.000002p-50 without relying on msvcrt precision limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml, wasm
Domain
compilers, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.