apache / apache/datafusion-comet
Build spark_size LargeList lengths as Int32 without Int64 cast
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow-up from #5233
`spark_size` for `LargeList` reuses Arrow's `length` kernel, which returns `Int64`, then `cast_with_options(..., Int32, safe: false)`, then (when nulls are present) `to_vec()` to patch null slots to `-1`. That is three allocations on the hot path.
On the `array_size` bench after #5233, `LargeList (10% null)` is only ~1.3x faster than `main` (7.54 µs vs 9.70 µs), while no-null `List` is ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself.
### Describe the potential solution
Build an `Int32` length array straight from the `LargeList` i64 offset buffer (`offsets[i+1] - offsets[i]`, with a checked conversion that errors on overflow rather than wrapping or becoming `-1`), then apply the same null →
`-1` rewrite as the List path (`null_count == 0` fast path; otherwise `into_parts` + `set_indices`).
Avoid allocating the intermediate Int64 `length` output and the Int32 cast result when those are only used to derive i32 sizes
### Additional context
- Bench coverage already exists: `spark_size: LargeList (10% null)` in `benches/array_size.rs`. Consider adding a no-null LargeList shape for parity with the List production path.
- Keep `safe: false` / checked overflow semantics: a length that does not fit in `i32` must error, not become null then `-1`
- Called out in review on #5233; left out of that PR so Map (#5266) and this path can land separately without blocking the List win
Contributor guide
Assessment
This issue has not been assessed yet.