apache / apache/datafusion-ballista
Q10 SF1000 regression after #2315: SortPreservingMergeExec exhausts fair memory pool under AQE default-on
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Describe the bug**
TPC-H Q10 at SF1000 fails under the AQE plan produced after #2315, hitting a memory-pool OOM in `SortPreservingMergeExec`. On the pre-#2315 baseline Q10 completed in ~53s on the same cluster and config; after #2315 it errors at runtime.
Full error:
```
DataFusion error: Arrow error: External error: Execution error:
Job 0380C6NW0A000 failed: Job failed due to stage 6 failed:
Task failed due to runtime execution error:
DataFusionError(Shared(External(DataFusionError(ResourcesExhausted(
"Failed to allocate additional 111.2 MB for SortPreservingMergeExec[0]
with 5.5 GB already allocated for this reservation
- 19.4 MB remain available for the total memory pool:
fair(pool_size: 5.6 GB)"
)))))
```
**To Reproduce**
- Ballista built from `f21c958` (merge commit of #2315)
- TPC-H SF1000, parquet/zstd, on Kubernetes: 32 executors × 8 concurrent tasks
- Memory pool: `fair` pool sized at 70% of a 64 GiB executor container ≈ 44.8 GB total, divided across 8 concurrent tasks → ~5.6 GB per-task budget (matches the error trace)
- AQE default-on (this PR flips it)
**Expected behavior**
Q10 should complete without OOM at this scale. It passed at ~53s on the same infrastructure before #2315.
**Additional context**
- Full 22-query run: 19 passed, Q10 regressed as above, Q21/Q22 hit unrelated cluster networking issues.
- Q11 was fixed by this PR (previously erroring, now 15.2s).
- Overall wall-clock improved ~12% (632s vs 721s baseline) despite the Q10 regression.
- The per-task pool sizing didn't change on our side — the new AQE plan for Q10 evidently needs more sort/merge memory than the previous plan. Possible causes: the removed redundant shuffle changing partition sizes, or an AQE-driven join/coalesce decision concentrating data.
- Original benchmark report: https://github.com/apache/datafusion-ballista/pull/2315#issuecomment-5308644923
Contributor guide
Research direction
Start by comparing the AQE plan introduced by #2315 with the pre-#2315 plan for TPC-H Q10, then inspect SortPreservingMergeExec's fair-pool reservation under the stated 32-executor and 8-task setup. Reproduce with SF1000 parquet/zstd and confirm that Q10 completes without the reported memory-pool OOM while preserving the AQE improvements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100