[Variant] `variant_to_arrow` union builder does not preserve `variant_get` conversion semantics
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
Here we use type-based info to determine which target the current variant should belong to. This seems incorrect
1. Variant::Int64(1) can't convert to DataType::Int8, but this should valid when calling `variant_get`
2. The logic in the current code (json -> variant, and variant spec), treats the exact number the same in Variant::Int64/Int32/Int16 etc(value has no strict type in variant) (like the `Equivalence Class` in parquet variant spec)
_Originally posted by @klion26 in https://github.com/apache/arrow-rs/pull/10313#discussion_r3841121078_
Contributor guide
Research direction
Start by tracing the variant_to_arrow union builder and compare its target-selection logic with the variant_get conversion semantics described in the issue. Review the json-to-variant behavior and the equivalence-class rules in the parquet variant spec; done means equivalent numeric values select compatible targets regardless of their Variant integer type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100