apache / apache/datafusion-comet
Slice arrays before normalizing to avoid allocating over the whole backing buffer
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the problem
Reported during review of #5262: reading two visible values out of a sliced array allocated approximately **8 MiB**, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes.
This is a component allocation measurement, not an end-to-end query timing.
### Describe the potential solution
Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected.
### Additional context
Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work.
Contributor guide
Research direction
Locate the array normalization and slicing entry points involved in the visible-range path, then reproduce the allocation described in the issue. Add a benchmark covering two visible values from a sliced array and verify that slicing first reduces allocation from the backing-buffer scale to the visible range.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100