microsoft / microsoft/powerbi-visuals-utils-typeutils
Double.roundToPrecision rounding issue
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 15
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
There is an issue with this line in Double.roundToPrecision
https://github.com/microsoft/powerbi-visuals-utils-typeutils/blob/c6485adf192418f8bb78982c5ee3050144659f99/src/double.ts#L379
I encountered this issue because the formattingutils use this function in the valueFormatter so that for a common configuration the formatting of e.g. 0.2 returns 0.19999999999999998. See this issue for details.
I found this thread which discusses different rounding to precision implementations and their caveats. The one that is working for me is this implementation
function roundToTwo(num) {
return +(Math.round(num + "e+2") + "e-2");
}
which I adjusted to this
function bestRound(x: number, precision: number) {
let p = Math.abs(precision);
let p1 = precision > 0 ? `-${p}` : `+${p}`;
let p2 = precision > 0 ? `+${p}` : `-${p}`;
return +(Math.round(x + `e${p1}` as any) + `e${p2}`);
}
although it's difficult to say if there are edge cases where it won't work. You can see my fix here. Let me know if I should send a pull request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/double.ts line 379 and inspect Double.roundToPrecision, then review the linked formattingutils issue and the proposed commit for the 0.2 reproduction. Compare the current behavior with the suggested rounding approach and investigate precision edge cases. Done means common values such as 0.2 no longer produce floating-point artifacts while precision rounding remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100