SedonaFairSpillPool can reject sort memory even when spill memory is available
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
We're seeing external sort failures where Sedona reports spillable memory is still available, but the allocation is rejected anyway.
The root cause appears to be that `SedonaFairSpillPool` splits the spill budget across all registered spill-capable consumers, including consumers that are idle and have reserved 0 bytes.
Minimal test case:
```rust
#[test]
fn test_idle_spillers_do_not_split_budget() {
let pool: Arc = Arc::new(SedonaFairSpillPool::new(100, 0.0));
let active_consumer = MemoryConsumer::new(active).with_can_spill(true);
let mut active = active_consumer.register(&pool);
let _idle_reservations: Vec<_> = (0..128)
.map(|i| {
MemoryConsumer::new(format!(idle-{i}))
.with_can_spill(true)
.register(&pool)
})
.collect();
active.try_grow(100).unwrap();
}
```
On current `main`, this fails with:
```text
Failed to allocate additional 100 bytes for active with 0 bytes already allocated - maximum available is 0 bytes.
Current unspillable memory usage: 0 bytes, spillable memory available: 100 bytes
```
Expected behavior: idle spill-capable consumers should not reduce the memory available to the active consumer.
Observed behavior: the active consumer effectively gets a 0-byte limit because the 100-byte pool is divided across 129 registered consumers, even though only one consumer is using memory.
This seems related to sort-heavy queries with many `ExternalSorter` consumers, where some sorters are idle but still reduce the budget for active sorters.
Contributor guide
Research direction
Start with the SedonaFairSpillPool implementation and the MemoryConsumer registration and allocation paths, using the minimal test case in the issue as the reproduction. Run the regression test with one active and many idle spill-capable consumers; done means the active consumer can allocate the full available pool and the test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 73/100