cockroachdb / cockroachdb/apd

Inconsistent (or buggy) Quantize behavior

Open
#122 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
805
Forks
48
PR merge metrics
No merged PRs in 30d

Description

Looks like `Quantize(result, amount, 0)` func works differently for:
- when `amount` is of the form `N.0*` (where N is integer number and N != 0)
- when `amount` is of the form `0.0*`

Here is a simple test to reproduce this (with `v3.1.0`):

```go
// quantize is a simple wrapper around `apd` library.
func quantize(amount *apd.Decimal, decimalPlaces uint32, roundingRule apd.Rounder) (*apd.Decimal, error) {
ctx := apd.BaseContext.WithPrecision(infinitePrecision)
ctx.Rounding = roundingRule

var amt apd.Decimal
exp := -int32(decimalPlaces)
_, err := ctx.Quantize(&amt, amount, exp)

return &amt, err
}

func TestQuantizeRounding(t *testing.T) {
// 1.1 -> 2 (works as expected)
got, err = quantize(apd.New(11, -1), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(2, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 1.01 -> 2 (works as expected)
got, err := quantize(apd.New(101, -2), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want := apd.New(2, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 0.1 -> 1 (works as expected)
got, err = quantize(apd.New(1, -1), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(1, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 0.01 -> 1 (doesn't work as expected! instead it does 0.01 -> 0 for some reason)
got, err = quantize(apd.New(1, -2), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(1, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Context.Quantize entry point and reproduce the supplied TestQuantizeRounding cases against v3.1.0, focusing on values with small fractional exponents and RoundUp. Done means the 0.01 to 1 case behaves consistently with the other examples without regressing the existing cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.