apache / apache/datafusion

Enable external operator reclaim / spill hooks for external memory managers

Open
#21,422 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?
issue: https://github.com/apache/datafusion-comet/issues/3873

Currently, datafusion comet cannot trigger datafusion native operators to spill in response to memory pressure from Spark’s task memory manager. When Spark task memory manager is under pressure, it may ask one consumer to spill so another consumer in the same task can make progress. Comet can route that request into native code, but there is currently no DataFusion interface for asking the spill-capable operator to reclaim memory in response to that external request.

### Describe the solution you'd like
Considering right now:
* We can add an interface for spill-capable operators to expose reclaim/spill behavior.
### Describe alternatives you've considered

* Current solution is basically only do local spills in df native and spark can make its own spark consumers spill like usual.
* Do not trigger spill here in df and instead have another signal we can send to df native.

### Additional context

* This may be generally useful for any spark accelerators where datafusion is embedded in a similar way to comet

Contributor guide

Open the contributing guide

Research direction

Start with the linked datafusion-comet issue and the described external memory-pressure flow. Determine how Spark task memory requests are routed into native code and identify the spill-capable DataFusion operators that would expose reclaim behavior. Done means an agreed interface lets external requests trigger operator reclaim or spill without breaking local spill behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.