cockroachdb / cockroachdb/apd

How to use Quantize with limited precision context

Open
#94 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
805
Forks
48
PR merge metrics
No merged PRs in 30d

Description

I'm building a financial application that deals with multiple currencies and multiple Contexts with settings appropriate to each currency. The DB storage supports the widest possible range of values to permit to store any currency, DB storage uses Decimal(78,18), i.e. `MaxExponent=60` and `MinExponent=-18` in `apd` terms.

I'm trying to quantize the value coming from DB with highest possible precision into the precision appropriate for given currency. The value is not large, but has many trailing decimal zeroes that I want to remove to bring it to correct precision.

And for whatever reason `Quantize()` does not produce the result I expect, but throws an `Overflow` condition instead. Below is the sample code that reproduces it:

```
package main

import (
"fmt"
"github.com/cockroachdb/apd/v2"
)

func main() {
dStr := "6410.000000000000000000" // small value with many trailing zeroes as seen in DB
wideCon := &apd.Context{ // widest supported context
MaxExponent: 78,
MinExponent: -18,
Traps: apd.DefaultTraps,
}
d, _, _ := wideCon.SetString(&apd.Decimal{}, dStr)

limitCon := &apd.Context{ // limited context suitable for given currency
Precision: 17,
MaxExponent: 9,
MinExponent: -8,
Traps: apd.DefaultTraps,
}
scratch := &apd.Decimal{}
con, err := limitCon.Quantize(scratch, d, limitCon.MinExponent) // attempt to quantize to correct precision, expect to work fine
fmt.Println(con, err, scratch) // fails with `overflow` condition
}
```

In this piece of code the `wideCon` is the widest supported context in the system, i.e. the DB default context, while `limitCon` is the target currency context. If I understand correctly the context settings - the `limitCon` effectively forces the values in this context to be within `999999999.99999999` range, which is exactly what I need.

The sample value is way below the `MaxExponent` limit, and I expect `limitCon.Quantize()` to produce same value with decimal places adjusted to fit, but instead it returns a `NaN` value and an `Overflow` condition.

The error goes away if I increase the `MaxExponent`, it works at `MaxExponent=12` for this value, but workarounding it this way means I will have to forfeit the ceiling check limit which I'd like to not do.

What is a better way to achieve what I need - round an arbitrary long precision value to a constrained precision settings force-rounded per currency and with correct ceiling constraint?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.