apache / apache/datafusion-comet

Slice arrays before normalizing to avoid allocating over the whole backing buffer

Open
#5,708 1 comment 0 reactions 0 assignees View on GitHub
area:expressions enhancement priority:medium requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the problem

Reported during review of #5262: reading two visible values out of a sliced array allocated approximately **8 MiB**, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.

This is a component allocation measurement, not an end-to-end query timing.

### Describe the potential solution

Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.

### Additional context

Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.

Contributor guide

Open the contributing guide

Research direction

Locate the array normalization and slicing entry points involved in the visible-range path, then reproduce the allocation described in the issue. Add a benchmark covering two visible values from a sliced array and verify that slicing first reduces allocation from the backing-buffer scale to the visible range.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.