apache / apache/datafusion

Dynamic BufferExec sizing: row limit + memory cap for sort pushdown

Open
#21,440 0 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge?**

#21426 introduced a configurable fixed-size `BufferExec` capacity (default 1GB) for sort pushdown. While this is better than the `SortExec` it replaces (which buffers the entire partition), a fixed size is not optimal for all cases:

- **Wide rows** (many columns, large strings): 1GB might not be enough row groups
- **Narrow rows** (few small columns): 1GB buffers far more data than needed

As noted by @alamb in https://github.com/apache/datafusion/pull/21426#discussion_r3045025814:

> I suspect a better solution than a fixed size buffer would be some calculation based on the actual size of the data (e.g. the number of rows to buffer). However, that is tricky to compute / constrain memory when large strings are involved. We probably would need to have both a row limit and a memory cap and pick the smaller of the two.

**Describe the solution you'd like**

Replace the fixed capacity with a dual-limit approach:

```
BufferExec stops buffering when EITHER limit is reached:
- Row limit: e.g., 100K rows (prevents over-buffering narrow rows)
- Memory cap: e.g., 1GB (prevents OOM for wide rows)
```

This adapts to different row widths automatically:
- Narrow rows (100 bytes/row): row limit triggers at ~10MB
- Wide rows (10KB/row): memory cap triggers at 1GB

**Related issues:**
- #21426 — Make BufferExec capacity configurable (current fixed-size approach)
- #21417 — Original issue for configurable buffer
- #21182 — Sort pushdown phase 2 (introduced BufferExec)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.