apache / apache/arrow-rs

`PushBuffers::clear_ranges` is quadratic and leaks

Open
#9,695 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**

`PushBuffers::clear_ranges` performs an O(N*M) scan to release consumed buffers, where N is the number of buffered ranges and M is the number of ranges to clear. On wide schemas (10k+ columns), this produces quadratic overhead in `PushDecoder` row group construction.

Additionally, `clear_ranges` matches buffers by exact range equality. When the IO layer coalesces adjacent requested ranges into fewer, larger fetches, the coalesced buffer never exactly matches any individual requested range, so `clear_ranges` silently skips it. The buffer leaks in `PushBuffers` until the decoder finishes or the caller manually calls `release_all_ranges`, increasing peak RSS proportionally to the amount of data coalesced ahead of the current row group.

This puts coalescing in a bind: without it, buffer count scales with range count and the quadratic `clear_ranges` dominates. With it, memory is not reclaimed incrementally.

**To Reproduce**

Use `PushDecoder` on a Parquet file with a wide schema (10k+ columns). Push data without coalesced buffers. Observe row group construction time scaling quadratically with column count. Conversely if using coalescing, observe memory not being released by `clear_ranges`, growing RSS until the decoder finishes.

**Expected behavior**

Buffer release should scale with buffer count (not range count), and coalesced or arbitrarily-sized buffers should be released incrementally as the decoder progresses.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with PushBuffers::clear_ranges and trace how PushDecoder constructs row groups and tracks buffered ranges. Reproduce the wide-schema and coalesced-buffer cases, then verify that release work scales with buffer count and that coalesced buffers are reclaimed incrementally.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.