[K8S][CORE] Heartbeat-timeout executor replacement may be classified as ExecutorKilled and skip task failure counting
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Version**
spark 3.5+ (k8s)
**Description**
I observed a behavior difference between Spark on Kubernetes and YARN for executor heartbeat-timeout scenarios.
In heartbeat timeout handling, HeartbeatReceiver triggers both:
1. sc.killAndReplaceExecutor(executorId) (which uses countFailures=true)
2. RemoveExecutor(executorId, ExecutorProcessLost(timeout))
However, in Kubernetes backend, doKillExecutors() historically removes executors eagerly with ExecutorKilled.
This can win the race against the later timeout-based RemoveExecutor(...ExecutorProcessLost...), so the final loss reason becomes ExecutorKilled.
Since ExecutorKilled is treated as non-app-caused failure (exitCausedByApp=false), task failures may not be counted towards spark.task.maxFailures, and fast-fail is less likely to happen.
On k8s, heartbeat timeouts can lead to repeated task retries without triggering fast-fail.
On yarn,the same task reaches the retry limit (4 attempts) and the application fails fast.
**Impact**
1. Heartbeat-timeout task failures may bypass failure counting.
2. Jobs can retry for a long time instead of fast-failing.
3. Behavior diverges from YARN under similar failure conditions.
**Suggested fix direction**
In K8s doKillExecutors, preserve countFailures semantics from killExecutors:
Keep ExecutorKilled for explicit driver-initiated non-counting kills.
For replacement kills (countFailures=true), avoid eagerly converting to ExecutorKilled; keep a non-ExecutorKilled reason (or allow timeout reason to land) so task-failure counting semantics are preserved.
If this direction is acceptable to maintainers, I can contribute a PR with a minimal fix and corresponding unit tests.
**Additional question:**
Could maintainers advise how to obtain an Apache JIRA account for new contributors?
Contributor guide
Research direction
Trace HeartbeatReceiver through sc.killAndReplaceExecutor, RemoveExecutor, ExecutorProcessLost, and ExecutorKilled, then inspect Kubernetes doKillExecutors and the existing killExecutors path. Reproduce the heartbeat-timeout flow and add focused unit tests for replacement versus explicit kills. Done means replacement losses preserve task-failure counting semantics while explicit non-counting kills remain ExecutorKilled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, scala, spark
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100