apache / apache/datafusion-ballista

Q10 SF1000 regression after #2315: SortPreservingMergeExec exhausts fair memory pool under AQE default-on

Open
#2,321 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Describe the bug**

TPC-H Q10 at SF1000 fails under the AQE plan produced after #2315, hitting a memory-pool OOM in `SortPreservingMergeExec`. On the pre-#2315 baseline Q10 completed in ~53s on the same cluster and config; after #2315 it errors at runtime.

Full error:
```
DataFusion error: Arrow error: External error: Execution error:
Job 0380C6NW0A000 failed: Job failed due to stage 6 failed:
Task failed due to runtime execution error:
DataFusionError(Shared(External(DataFusionError(ResourcesExhausted(
"Failed to allocate additional 111.2 MB for SortPreservingMergeExec[0]
with 5.5 GB already allocated for this reservation
- 19.4 MB remain available for the total memory pool:
fair(pool_size: 5.6 GB)"
)))))
```

**To Reproduce**

- Ballista built from `f21c958` (merge commit of #2315)
- TPC-H SF1000, parquet/zstd, on Kubernetes: 32 executors × 8 concurrent tasks
- Memory pool: `fair` pool sized at 70% of a 64 GiB executor container ≈ 44.8 GB total, divided across 8 concurrent tasks → ~5.6 GB per-task budget (matches the error trace)
- AQE default-on (this PR flips it)

**Expected behavior**

Q10 should complete without OOM at this scale. It passed at ~53s on the same infrastructure before #2315.

**Additional context**

- Full 22-query run: 19 passed, Q10 regressed as above, Q21/Q22 hit unrelated cluster networking issues.
- Q11 was fixed by this PR (previously erroring, now 15.2s).
- Overall wall-clock improved ~12% (632s vs 721s baseline) despite the Q10 regression.
- The per-task pool sizing didn't change on our side — the new AQE plan for Q10 evidently needs more sort/merge memory than the previous plan. Possible causes: the removed redundant shuffle changing partition sizes, or an AQE-driven join/coalesce decision concentrating data.
- Original benchmark report: https://github.com/apache/datafusion-ballista/pull/2315#issuecomment-5308644923

Contributor guide

Open the contributing guide

Research direction

Start by comparing the AQE plan introduced by #2315 with the pre-#2315 plan for TPC-H Q10, then inspect SortPreservingMergeExec's fair-pool reservation under the stated 32-executor and 8-task setup. Reproduce with SF1000 parquet/zstd and confirm that Q10 completes without the reported memory-pool OOM while preserving the AQE improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.