microsoft / microsoft/powerbi-visuals-utils-typeutils

Double.roundToPrecision rounding issue

Open
#34 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
15
Forks
30
PR merge metrics
No merged PRs in 30d

Description

There is an issue with this line in Double.roundToPrecision
https://github.com/microsoft/powerbi-visuals-utils-typeutils/blob/c6485adf192418f8bb78982c5ee3050144659f99/src/double.ts#L379
I encountered this issue because the formattingutils use this function in the valueFormatter so that for a common configuration the formatting of e.g. 0.2 returns 0.19999999999999998. See this issue for details.
I found this thread which discusses different rounding to precision implementations and their caveats. The one that is working for me is this implementation

function roundToTwo(num) {    
    return +(Math.round(num + "e+2")  + "e-2");
}

which I adjusted to this

function bestRound(x: number, precision: number) {
  let p = Math.abs(precision);
  let p1 = precision > 0 ? `-${p}` : `+${p}`;
  let p2 = precision > 0 ? `+${p}` : `-${p}`;
  return +(Math.round(x + `e${p1}` as any)  + `e${p2}`);
}

although it's difficult to say if there are edge cases where it won't work. You can see my fix here. Let me know if I should send a pull request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/double.ts line 379 and inspect Double.roundToPrecision, then review the linked formattingutils issue and the proposed commit for the 0.2 reproduction. Compare the current behavior with the suggested rounding approach and investigate precision edge cases. Done means common values such as 0.2 no longer produce floating-point artifacts while precision rounding remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.