apache / apache/spark

[DISCUSS][CORE] Should managed off-heap memory take the container limit into account?

Open
#57,530 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

I have been looking at how Spark managed off-heap memory behaves in a memory-limited container.
One thing that seems easy to misconfigure is that these three limits are independent:

```text
container limit: 1500 MiB
JVM max heap: 1024 MiB
spark.memory.offHeap.size: 2048 MiB
```

`spark.memory.offHeap.size` is only an upper bound used by Spark's memory manager, so the example
above does not fail immediately. The problem is that `UnifiedMemoryManager` does not know about the
1500 MiB process limit. If heap, managed off-heap memory, direct buffers, thread stacks, and other
native allocations grow at the same time, the container can kill the executor before Spark gets a
chance to spill.

In that case there may be no useful Java OOM. From outside the container it can just look like:

```text
exit code 137
OOMKilled=true
```

Using JVM heap occupancy as the feedback signal does not seem right here. It cannot see off-heap
or other native memory, and it moves with GC. The cgroup usage and limit describe the boundary that
actually kills the process.

I made a small prototype that reads cgroup v1/v2 values and uses them to put a lower admission
limit on Spark-managed off-heap memory. At startup the rough budget is:

```text
cgroup limit - JVM max heap - native reserve
```

The admission limit can then be reduced when total cgroup usage is high. This still relies on the
existing `MemoryConsumer` and spill code; it does not add another eviction mechanism.

There are some non-obvious details. In particular, setting the limit to zero is not safe because a
task may fail to allocate its first sorter page and throw `SparkOutOfMemoryError` instead of
spilling. Also, a fixed native reserve is probably too simplistic across different collectors,
core counts, and deployment modes.

Before spending more time on the prototype, I would like to check the expected direction:

1. Should Spark use the container limit when admitting managed off-heap allocations, or is this
expected to be handled entirely by executor memory-overhead configuration?
2. If Spark should handle it, would a startup warning/static cap be preferable to runtime
cgroup-pressure feedback?

If this belongs in Spark Core, I can follow up with a focused reproducer and tests for the approach
maintainers prefer.

Contributor guide

Open the contributing guide

Research direction

Start with UnifiedMemoryManager and the existing MemoryConsumer spill path, then review the prototype's cgroup v1/v2 readings and JVM/native budget assumptions. A useful outcome is a maintainer-approved direction—executor-overhead configuration, a startup cap or warning, or runtime feedback—followed by a focused reproducer and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.