arrow-row: expose `decoded_type` API so downstreams can compute row-format round-trip type expectations
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
`RowConverter` accepts types on encode whose **decode output type differs from the encode input type**: dictionaries (and dictionary-containing nested types) are flattened to their value type on decode via the internal `corrected_type` step. For example:
- `Dictionary(Int32, Utf8)` decodes as `Utf8`
- `ListView(Dictionary(Int32, Utf8))` decodes as `ListView(Utf8)`
- `List(Dictionary(Int32, Utf8))` decodes as `List(Utf8)`
This mapping is arrow-row internal knowledge with **no public API to query it**. Downstreams that need to round-trip arrays through the row format (e.g. DataFusion's group-by, which stores group keys as rows and re-emits them as arrays) must mirror decoder internals by hand to know which output types to expect and reconstruct.
This has produced two real bugs of the same class:
1. #10413 — `decode_fixed_size_list` diverged from the other list-like decoders by *not* applying the correction, panicking on `FixedSizeList` (fixed by #10414).
2. [apache/datafusion#23523 review](https://github.com/apache/datafusion/pull/23523#discussion_r3643717742) — DataFusion accepted `ListView` for its rows-backed group storage but missed the reconstruction arm for view-lists, silently emitting `ListView(Utf8)` where the schema declared `ListView(Dictionary(Int32, Utf8))`.
Both stem from the same gap: each downstream has to re-derive, per container type, "what type will decode hand me back?" — and every missed container is a bug.
**Describe the solution you'd like**
Expose the mapping as a public API, e.g.:
```rust
/// Returns the data type produced by `RowConverter::convert_rows` for
/// an input column of `data_type` — identical to the input except that
/// dictionary-encoded values (at any nesting level) are replaced by
/// their value type.
pub fn decoded_type(data_type: &DataType) -> DataType
```
(placement flexible: free function in `arrow_row`, method on `SortField`, or `RowConverter::decoded_types(&self) -> Vec`).
Together with a **property test in arrow-row** asserting the contract for every supported type shape:
```rust
for dt in representative_supported_shapes() {
let array = make_test_array(&dt);
let rows = converter.convert_columns(&[array])?;
let decoded = converter.convert_rows(&rows)?;
assert_eq!(decoded[0].data_type(), &decoded_type(&dt));
}
```
The invariant test is arguably the bigger win: it would have caught #10413 automatically (panic = contract violation), and prevents any future decoder from silently diverging from the others the way `decode_fixed_size_list` did.
With this API, downstream code reduces to a mechanical rule: `decoded_type(t) == t` → array round-trips as-is; otherwise recurse over the structure and re-encode exactly where the two types differ.
**Describe alternatives you've considered**
- **A boolean `round_trips_losslessly(&DataType) -> bool`** — simpler, but only says *whether* the type changes, not *what comes back*, so downstreams still can't derive reconstruction mechanically.
- **Making decode preserve dictionary types** — would eliminate the mapping entirely, but re-encoding dictionaries on decode is expensive, changes long-standing semantics, and the flattened output is often what callers want.
- **Status quo** — downstreams mirror decoder internals; two bugs in two months suggests this doesn't scale.
**Additional context**
- #10413 / #10414 (the `decode_fixed_size_list` divergence)
- [apache/datafusion#23523](https://github.com/apache/datafusion/pull/23523) — the DataFusion PR where the second instance surfaced, with an end-to-end reproducer in the review thread
Happy to implement this (API + property test) if the direction sounds good.
Contributor guide
Assessment
This issue has not been assessed yet.