apache / apache/arrow-rs

arrow-row: expose `decoded_type` API so downstreams can compute row-format round-trip type expectations

Open
#10,427 2 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

`RowConverter` accepts types on encode whose **decode output type differs from the encode input type**: dictionaries (and dictionary-containing nested types) are flattened to their value type on decode via the internal `corrected_type` step. For example:

- `Dictionary(Int32, Utf8)` decodes as `Utf8`
- `ListView(Dictionary(Int32, Utf8))` decodes as `ListView(Utf8)`
- `List(Dictionary(Int32, Utf8))` decodes as `List(Utf8)`

This mapping is arrow-row internal knowledge with **no public API to query it**. Downstreams that need to round-trip arrays through the row format (e.g. DataFusion's group-by, which stores group keys as rows and re-emits them as arrays) must mirror decoder internals by hand to know which output types to expect and reconstruct.

This has produced two real bugs of the same class:

1. #10413 — `decode_fixed_size_list` diverged from the other list-like decoders by *not* applying the correction, panicking on `FixedSizeList` (fixed by #10414).
2. [apache/datafusion#23523 review](https://github.com/apache/datafusion/pull/23523#discussion_r3643717742) — DataFusion accepted `ListView` for its rows-backed group storage but missed the reconstruction arm for view-lists, silently emitting `ListView(Utf8)` where the schema declared `ListView(Dictionary(Int32, Utf8))`.

Both stem from the same gap: each downstream has to re-derive, per container type, "what type will decode hand me back?" — and every missed container is a bug.

**Describe the solution you'd like**

Expose the mapping as a public API, e.g.:

```rust
/// Returns the data type produced by `RowConverter::convert_rows` for
/// an input column of `data_type` — identical to the input except that
/// dictionary-encoded values (at any nesting level) are replaced by
/// their value type.
pub fn decoded_type(data_type: &DataType) -> DataType
```

(placement flexible: free function in `arrow_row`, method on `SortField`, or `RowConverter::decoded_types(&self) -> Vec`).

Together with a **property test in arrow-row** asserting the contract for every supported type shape:

```rust
for dt in representative_supported_shapes() {
let array = make_test_array(&dt);
let rows = converter.convert_columns(&[array])?;
let decoded = converter.convert_rows(&rows)?;
assert_eq!(decoded[0].data_type(), &decoded_type(&dt));
}
```

The invariant test is arguably the bigger win: it would have caught #10413 automatically (panic = contract violation), and prevents any future decoder from silently diverging from the others the way `decode_fixed_size_list` did.

With this API, downstream code reduces to a mechanical rule: `decoded_type(t) == t` → array round-trips as-is; otherwise recurse over the structure and re-encode exactly where the two types differ.

**Describe alternatives you've considered**

- **A boolean `round_trips_losslessly(&DataType) -> bool`** — simpler, but only says *whether* the type changes, not *what comes back*, so downstreams still can't derive reconstruction mechanically.
- **Making decode preserve dictionary types** — would eliminate the mapping entirely, but re-encoding dictionaries on decode is expensive, changes long-standing semantics, and the flattened output is often what callers want.
- **Status quo** — downstreams mirror decoder internals; two bugs in two months suggests this doesn't scale.

**Additional context**

- #10413 / #10414 (the `decode_fixed_size_list` divergence)
- [apache/datafusion#23523](https://github.com/apache/datafusion/pull/23523) — the DataFusion PR where the second instance surfaced, with an end-to-end reproducer in the review thread

Happy to implement this (API + property test) if the direction sounds good.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.