apache / apache/datafusion-comet

Spark cannot reclaim memory from native operators (spill callback returns 0)

Open
#3,873 4 comments 0 reactions 2 assignees Claimed by @nathanb9 View on GitHub
performance priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

## Describe the bug

**note: this issue was generated with AI and needs verification**

When Spark's memory manager is under pressure and calls `spill()` on Comet's `NativeMemoryConsumer`, it returns 0 — meaning Spark cannot reclaim any memory from Comet's native operators. This prevents cross-task memory eviction and causes Comet to require significantly more off-heap memory than necessary at scale.

From `CometTaskMemoryManager.java`:

```java
private class NativeMemoryConsumer extends MemoryConsumer {
public long spill(long size, MemoryConsumer trigger) {
return 0; // No spilling
}
}
```

## Impact

When multiple concurrent tasks share a constrained off-heap pool (e.g., 16 tasks sharing 16GB), each task's shuffle writer greedily buffers data until `try_grow()` fails. Since Spark cannot reclaim memory from any of them, tasks compete for the shared pool without coordination, leading to OOM at lower memory settings.

Benchmarking TPC-H SF100 Q9 with `local[4]` showed Comet's memory growing elastically with the offHeap budget (448 MB increase from 4g to 8g), while Spark's native operators stay flat because they participate in the spill protocol.

## Expected behavior

`NativeMemoryConsumer.spill()` should signal the native memory pool to apply backpressure, causing DataFusion operators (Sort, Aggregate, Shuffle) to spill their internal state to disk. The actual bytes freed should be returned to Spark.

## Proposed approach

Add a `SpillState` struct with atomics and a condvar for cross-thread coordination:

1. When Spark calls `spill(size)`, JNI into native to set spill pressure
2. The pool's `try_grow()` checks pressure and returns `ResourcesExhausted`
3. DataFusion operators catch this and spill internally
4. As operators call `shrink()`, freed bytes are tracked and returned to Spark

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.