apache / apache/datafusion-comet

Build spark_size LargeList lengths as Int32 without Int64 cast

Open
#5,272 1 comment 0 reactions 1 assignee Claimed by @0lai0 View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow-up from #5233
`spark_size` for `LargeList` reuses Arrow's `length` kernel, which returns `Int64`, then `cast_with_options(..., Int32, safe: false)`, then (when nulls are present) `to_vec()` to patch null slots to `-1`. That is three allocations on the hot path.

On the `array_size` bench after #5233, `LargeList (10% null)` is only ~1.3x faster than `main` (7.54 µs vs 9.70 µs), while no-null `List` is ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself.

### Describe the potential solution

Build an `Int32` length array straight from the `LargeList` i64 offset buffer (`offsets[i+1] - offsets[i]`, with a checked conversion that errors on overflow rather than wrapping or becoming `-1`), then apply the same null →
`-1` rewrite as the List path (`null_count == 0` fast path; otherwise `into_parts` + `set_indices`).

Avoid allocating the intermediate Int64 `length` output and the Int32 cast result when those are only used to derive i32 sizes

### Additional context
- Bench coverage already exists: `spark_size: LargeList (10% null)` in `benches/array_size.rs`. Consider adding a no-null LargeList shape for parity with the List production path.
- Keep `safe: false` / checked overflow semantics: a length that does not fit in `i32` must error, not become null then `-1`
- Called out in review on #5233; left out of that PR so Map (#5266) and this path can land separately without blocking the List win

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.