cockroachdb / cockroachdb/apd

How to use Quantize with limited precision context

Offen
#94 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
805
Forks
48
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

I'm building a financial application that deals with multiple currencies and multiple Contexts with settings appropriate to each currency. The DB storage supports the widest possible range of values to permit to store any currency, DB storage uses Decimal(78,18), i.e. `MaxExponent=60` and `MinExponent=-18` in `apd` terms.

I'm trying to quantize the value coming from DB with highest possible precision into the precision appropriate for given currency. The value is not large, but has many trailing decimal zeroes that I want to remove to bring it to correct precision.

And for whatever reason `Quantize()` does not produce the result I expect, but throws an `Overflow` condition instead. Below is the sample code that reproduces it:

```
package main

import (
"fmt"
"github.com/cockroachdb/apd/v2"
)

func main() {
dStr := "6410.000000000000000000" // small value with many trailing zeroes as seen in DB
wideCon := &apd.Context{ // widest supported context
MaxExponent: 78,
MinExponent: -18,
Traps: apd.DefaultTraps,
}
d, _, _ := wideCon.SetString(&apd.Decimal{}, dStr)

limitCon := &apd.Context{ // limited context suitable for given currency
Precision: 17,
MaxExponent: 9,
MinExponent: -8,
Traps: apd.DefaultTraps,
}
scratch := &apd.Decimal{}
con, err := limitCon.Quantize(scratch, d, limitCon.MinExponent) // attempt to quantize to correct precision, expect to work fine
fmt.Println(con, err, scratch) // fails with `overflow` condition
}
```

In this piece of code the `wideCon` is the widest supported context in the system, i.e. the DB default context, while `limitCon` is the target currency context. If I understand correctly the context settings - the `limitCon` effectively forces the values in this context to be within `999999999.99999999` range, which is exactly what I need.

The sample value is way below the `MaxExponent` limit, and I expect `limitCon.Quantize()` to produce same value with decimal places adjusted to fit, but instead it returns a `NaN` value and an `Overflow` condition.

The error goes away if I increase the `MaxExponent`, it works at `MaxExponent=12` for this value, but workarounding it this way means I will have to forfeit the ceiling check limit which I'd like to not do.

What is a better way to achieve what I need - round an arbitrary long precision value to a constrained precision settings force-rounded per currency and with correct ceiling constraint?

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.