apache / apache/arrow-rs

Support `Decimal128` / `Decimal256` in json inference

Open
#4,856 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm processing jsons with some exceptionally large numbers provided as json `number`. They exceed both `int64` and exactly represented `float64`.

Currently json inference fallback to `float64`, which causes reader to lose exactness when parsing values.

**Describe the solution you'd like**
I think since Rust `arrow` now supports decimal data type, it could be used also for json inference (parsing already works if I enforce a proper data type).

I'm not sure which precision/scale should be inferred in this case, possibly inference should track necessary scale and assume `(max precision - necessary scale, necessary scale)`.

**Describe alternatives you've considered**
One way to work around that would be to use strings, since `json::ReaderBuilder` with `.with_coerce_primitive(true)` allows to parse interchangebly strings and numbers.
This might actually still be a valid solution if we need to support numbers that are beyond even `decimal256`.

**Additional context**
```rust
const DATA_BIG_NUM: &str = r#"{"x": 1, "y": "01"}
{"x": 2000000000, "y": "02"}
{"x": 102174585645880460651957762, "y": "03"}
{"x": 102174585645880460651957762102, "y": "04"}"#;

#[test]
fn test_json_infers_coerce() {
let mut cursor = std::io::Cursor::new(&DATA_BIG_NUM);
let schema = infer_json_schema_from_seekable(&mut cursor, Some(3)).unwrap();
let mut json = json::ReaderBuilder::new(schema.into()).build(cursor).unwrap();

let batch = json.next().unwrap().unwrap();
// Inference falls into float64 datatype, which is lossy as demonstrated below
assert_eq!(DataType::Decimal128(30, 0), *batch.schema().fields()[0].data_type());
assert_eq!(
102174585645880460651957762i128,
Decimal128Array::from(batch.column(0).to_data()).value(2)
);
}
```

Contributor guide

Open the contributing guide

Research direction

Start at the infer_json_schema_from_seekable and JSON ReaderBuilder entry points, using the DATA_BIG_NUM example and test_json_infers_coerce reproduction. Determine the precision and scale policy for Decimal128 and Decimal256, including values too large for those types. Done means inference preserves exact values and tests cover the supported range and fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.