[DISCUSS][CORE] Should managed off-heap memory take the container limit into account?
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
I have been looking at how Spark managed off-heap memory behaves in a memory-limited container.
One thing that seems easy to misconfigure is that these three limits are independent:
```text
container limit: 1500 MiB
JVM max heap: 1024 MiB
spark.memory.offHeap.size: 2048 MiB
```
`spark.memory.offHeap.size` is only an upper bound used by Spark's memory manager, so the example
above does not fail immediately. The problem is that `UnifiedMemoryManager` does not know about the
1500 MiB process limit. If heap, managed off-heap memory, direct buffers, thread stacks, and other
native allocations grow at the same time, the container can kill the executor before Spark gets a
chance to spill.
In that case there may be no useful Java OOM. From outside the container it can just look like:
```text
exit code 137
OOMKilled=true
```
Using JVM heap occupancy as the feedback signal does not seem right here. It cannot see off-heap
or other native memory, and it moves with GC. The cgroup usage and limit describe the boundary that
actually kills the process.
I made a small prototype that reads cgroup v1/v2 values and uses them to put a lower admission
limit on Spark-managed off-heap memory. At startup the rough budget is:
```text
cgroup limit - JVM max heap - native reserve
```
The admission limit can then be reduced when total cgroup usage is high. This still relies on the
existing `MemoryConsumer` and spill code; it does not add another eviction mechanism.
There are some non-obvious details. In particular, setting the limit to zero is not safe because a
task may fail to allocate its first sorter page and throw `SparkOutOfMemoryError` instead of
spilling. Also, a fixed native reserve is probably too simplistic across different collectors,
core counts, and deployment modes.
Before spending more time on the prototype, I would like to check the expected direction:
1. Should Spark use the container limit when admitting managed off-heap allocations, or is this
expected to be handled entirely by executor memory-overhead configuration?
2. If Spark should handle it, would a startup warning/static cap be preferable to runtime
cgroup-pressure feedback?
If this belongs in Spark Core, I can follow up with a focused reproducer and tests for the approach
maintainers prefer.
Contributor guide
Research direction
Start with UnifiedMemoryManager and the existing MemoryConsumer spill path, then review the prototype's cgroup v1/v2 readings and JVM/native budget assumptions. A useful outcome is a maintainer-approved direction—executor-overhead configuration, a startup cap or warning, or runtime feedback—followed by a focused reproducer and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100