apache / apache/arrow-rs

`arrow-json` primitive decoder never detects numeric-string/fractional-float type conflicts, so `ignore_type_conflicts` can't act on them

Open
#10,265 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

### Describe the bug

When decoding a value for a numeric column, `arrow-json` should treat the JSON literal `1234` differently from the JSON literal `"1234"`: one is a number token, the other is a string token that contains digits.
It does not: both tokens are accepted and produce the identical `Int64` value `1234`. The decoder parses the *contents* of a string as if it were a number instead of first checking whether the token was a number in the
first place — so no type conflict is ever raised for this case.

The option that should govern this is `with_ignore_type_conflicts`. Its doc comment describes precisely this scenario:

> if the type is declared to be ... `DataType::Int32` but the reader
> encounters a string value `"foo"` ... `false` (the default): The reader
> will return an error.

But that guarantee only holds when the string's contents fail to parse as a number (`"foo"`). When the string's contents *do* parse as a number (`"42"`), no conflict is detected at all, so `ignore_type_conflicts` never gets a chance to act — the value is silently accepted regardless of whether `ignore_type_conflicts` is `true` or `false`.

The same gap exists one level down: a JSON **float** with a fractional part (`5.7`) decoded into an `Int64` column is silently **truncated** to `5` rather than being treated as a conflict — again irrespective of
`ignore_type_conflicts`. This is silent data loss, not just a lenient alternate encoding.

### To Reproduce

```rust
use arrow::array::AsArray;
use arrow::datatypes::{DataType, Field, Schema};
use arrow::json::StructMode;
use arrow::json::reader::ReaderBuilder;
use std::sync::Arc;

fn decode_one(schema: Arc, json_row: &str, ignore_type_conflicts: bool) -> Result {
let mut decoder = ReaderBuilder::new(schema.clone())
.with_struct_mode(StructMode::ListOnly)
.with_ignore_type_conflicts(ignore_type_conflicts)
.build_decoder()
.map_err(|e| e.to_string())?;
decoder.decode(json_row.as_bytes()).map_err(|e| e.to_string())?;
decoder.decode(b"\n").map_err(|e| e.to_string())?;
let batch = decoder.flush().map_err(|e| e.to_string())?.unwrap();
Ok(batch.column(0).as_primitive::().value(0))
}

fn main() {
let schema = Arc::new(Schema::new(vec![Field::new("AGE", DataType::Int64, true)]));

// Decode a JSON string "42" into an Int64 column, with the default
// ignore_type_conflicts=false.
println!("{:?}", decode_one(schema.clone(), r#"["42"]"#, false));

// Decode a JSON float 5.7 into an Int64 column, with the default
// ignore_type_conflicts=false.
println!("{:?}", decode_one(schema.clone(), r#"[5.7]"#, false));
}
```

Output on 59.0.0:

```
Ok(42)
Ok(5)
```

The same holds for `Float32`/`Float64` targets given a numeric JSON string
(`"177.8"` → `Float32` value `177.8`, `"72.5"` → `Float64` value `72.5`),
confirmed with the same harness.

### Expected behavior

With `ignore_type_conflicts` at its default (`false`), both calls should return `Err`, not `Ok`:

- `r#"["42"]"#` into an `Int64` column: the token is a JSON string, not a number, so this should be rejected as a type conflict — the same as any other string-into-number mismatch (e.g. `"foo"`, which *is* correctly
rejected today).
- `r#"[5.7]"#` into an `Int64` column: `5.7` is not representable as an `Int64` without loss, so this should be rejected as a type conflict rather than silently truncated to `5`.

### Additional context

Both are silent acceptance of malformed input rather than a decode error.

Contributor guide

Open the contributing guide

Research direction

Start with the arrow-json ReaderBuilder and its build_decoder path, then reproduce the two cases from the issue: a numeric string into Int64 and 5.7 into Int64, with ignore_type_conflicts disabled. Trace the primitive decoding and existing conflict tests; done means both cases return errors by default, while the option's intended behavior remains covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.