cockroachdb / cockroachdb/apd

Inconsistent (or buggy) Quantize behavior

Aperta
#122 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
805
Fork
48
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Looks like `Quantize(result, amount, 0)` func works differently for:
- when `amount` is of the form `N.0*` (where N is integer number and N != 0)
- when `amount` is of the form `0.0*`

Here is a simple test to reproduce this (with `v3.1.0`):

```go
// quantize is a simple wrapper around `apd` library.
func quantize(amount *apd.Decimal, decimalPlaces uint32, roundingRule apd.Rounder) (*apd.Decimal, error) {
ctx := apd.BaseContext.WithPrecision(infinitePrecision)
ctx.Rounding = roundingRule

var amt apd.Decimal
exp := -int32(decimalPlaces)
_, err := ctx.Quantize(&amt, amount, exp)

return &amt, err
}

func TestQuantizeRounding(t *testing.T) {
// 1.1 -> 2 (works as expected)
got, err = quantize(apd.New(11, -1), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(2, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 1.01 -> 2 (works as expected)
got, err := quantize(apd.New(101, -2), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want := apd.New(2, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 0.1 -> 1 (works as expected)
got, err = quantize(apd.New(1, -1), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(1, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}

// 0.01 -> 1 (doesn't work as expected! instead it does 0.01 -> 0 for some reason)
got, err = quantize(apd.New(1, -2), 0, apd.RoundUp)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want = apd.New(1, 0)
if want.CmpTotal(got) != 0 {
t.Fatalf("want: %+v, got: %+v", want, got)
}
}
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.