grafana / grafana/google-bigquery-datasource
NUMERIC / BIGNUMERIC values lose precision (converted via float64)
- Dominant language
- TypeScript
- Stars
- 40
- Forks
- 24
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 8
Description
### Summary
`NUMERIC` and `BIGNUMERIC` columns are returned through a `float64` conversion, so values with more than ~15–16 significant digits are silently rounded/corrupted. `NUMERIC` is an exact decimal (38 digits, scale 9) and `BIGNUMERIC` is 76 digits, but `float64` has only ~15–16 significant decimal digits.
### Location
`pkg/bigquery/driver/utils.go`:
```go
case "NUMERIC", "BIGNUMERIC":
conv, _ := v.(*big.Rat).Float64()
return conv, nil
```
The exact `*big.Rat` is downcast to `float64` (and the `exact` bool returned by `Float64()` is discarded).
### Evidence
Querying these through the plugin and comparing the returned value to the exact input:
| query | exact value | returned by plugin |
|---|---|---|
| `CAST('12345678901234567.123456789' AS NUMERIC)` | 12345678901234567.123456789 | `12345678901234568` |
| `CAST('9999999999999999' AS NUMERIC)` | 9999999999999999 | `10000000000000000` |
| `CAST('1234567890123456789012345.123' AS BIGNUMERIC)` | 1234567890123456789012345.123 | `1.2345678901234568e+24` |
**Control:** casting the same NUMERIC to `STRING` returns it exactly (`12345678901234567.123456789`), confirming BigQuery holds the exact value and the loss is introduced by the plugin's `float64` conversion.
Note the second row: an exact 16-digit integer (`9999999999999999`) comes back as `10000000000000000` — off by one, not merely a fractional rounding.
### Impact
Financial amounts, large IDs, and high-precision measurements stored as `NUMERIC`/`BIGNUMERIC` are silently corrupted, with no error. The loss is unrecoverable downstream.
### Suggested fix
Return the exact value losslessly — e.g. as a string via `(*big.Rat).FloatString(scale)` using the column's scale — instead of `float64`.
This is a behavior change (the field would surface as string rather than number), so it may warrant gating or applying it only when the value can't be represented exactly. Happy to discuss the approach and open a PR.
Contributor guide
Research direction
Start in pkg/bigquery/driver/utils.go at the NUMERIC/BIGNUMERIC conversion case, then inspect how its returned value is exposed by the plugin. Reproduce the listed precision-loss examples and determine the agreed lossless representation, including the behavior change from numeric to string. Done means exact NUMERIC and BIGNUMERIC values no longer undergo silent float64 corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100