apache / apache/arrow-rs

[Variant] `variant_to_arrow` union builder does not preserve `variant_get` conversion semantics

Open
#10,905 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

Here we use type-based info to determine which target the current variant should belong to. This seems incorrect
1. Variant::Int64(1) can't convert to DataType::Int8, but this should valid when calling `variant_get`
2. The logic in the current code (json -> variant, and variant spec), treats the exact number the same in Variant::Int64/Int32/Int16 etc(value has no strict type in variant) (like the `Equivalence Class` in parquet variant spec)

_Originally posted by @klion26 in https://github.com/apache/arrow-rs/pull/10313#discussion_r3841121078_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the variant_to_arrow union builder and compare its target-selection logic with the variant_get conversion semantics described in the issue. Review the json-to-variant behavior and the equivalence-class rules in the parquet variant spec; done means equivalent numeric values select compatible targets regardless of their Variant integer type.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.