cockroachdb / cockroachdb/apd

RoundCeil(0.002, -1) != 0.1

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
805
Forks
48
PR merge metrics
No merged PRs in 30d

Description

When doing RoundCeiling operation to 0.002 with precision set to -1, we get 0 instead of 0.1.
However for RoundCeiling(1.002, -1) we get 1.1.

```
func TestDecimal_RoundCeil_Bug(t *testing.T) {
var num apd.Decimal
_, _, err := num.SetString("0.002")
if err != nil {
t.Fatal(err)
}
ctx := apd.Context{
Precision: 100,
Rounding: apd.RoundCeiling,
MaxExponent: 1000,
MinExponent: -1000,
Traps: apd.DefaultTraps,
}
_, err = ctx.Quantize(&num, &num, -1)
if err != nil {
t.Fatal(err)
}
if num.String() != "0.1" {
t.Errorf("Expected 0.1, got %s", num.String())
}
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.