Support `Decimal128` / `Decimal256` in json inference
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm processing jsons with some exceptionally large numbers provided as json `number`. They exceed both `int64` and exactly represented `float64`.
Currently json inference fallback to `float64`, which causes reader to lose exactness when parsing values.
**Describe the solution you'd like**
I think since Rust `arrow` now supports decimal data type, it could be used also for json inference (parsing already works if I enforce a proper data type).
I'm not sure which precision/scale should be inferred in this case, possibly inference should track necessary scale and assume `(max precision - necessary scale, necessary scale)`.
**Describe alternatives you've considered**
One way to work around that would be to use strings, since `json::ReaderBuilder` with `.with_coerce_primitive(true)` allows to parse interchangebly strings and numbers.
This might actually still be a valid solution if we need to support numbers that are beyond even `decimal256`.
**Additional context**
```rust
const DATA_BIG_NUM: &str = r#"{"x": 1, "y": "01"}
{"x": 2000000000, "y": "02"}
{"x": 102174585645880460651957762, "y": "03"}
{"x": 102174585645880460651957762102, "y": "04"}"#;
#[test]
fn test_json_infers_coerce() {
let mut cursor = std::io::Cursor::new(&DATA_BIG_NUM);
let schema = infer_json_schema_from_seekable(&mut cursor, Some(3)).unwrap();
let mut json = json::ReaderBuilder::new(schema.into()).build(cursor).unwrap();
let batch = json.next().unwrap().unwrap();
// Inference falls into float64 datatype, which is lossy as demonstrated below
assert_eq!(DataType::Decimal128(30, 0), *batch.schema().fields()[0].data_type());
assert_eq!(
102174585645880460651957762i128,
Decimal128Array::from(batch.column(0).to_data()).value(2)
);
}
```
Contributor guide
Research direction
Start at the infer_json_schema_from_seekable and JSON ReaderBuilder entry points, using the DATA_BIG_NUM example and test_json_infers_coerce reproduction. Determine the precision and scale policy for Decimal128 and Decimal256, including values too large for those types. Done means inference preserves exact values and tests cover the supported range and fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100