kubeflow / kubeflow/spark-operator

Fenced, progress-preserving driver restarts

Open
#3,040 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What feature you would like to be added?

An opt-in extension to `SparkApplication.spec.restartPolicy` that makes operator-driven restarts **fenced** and **progress-preserving**:

1. **Fenced:** before each re-submission, the operator atomically advances a monotonic *epoch* in an external state store (Redis in the initial implementation; a small pluggable interface for others). Writes guarded by an older epoch are atomically rejected, so a zombie of the previous driver — still running on a node the kubelet can no longer report on — can never commit output again.
2. **Progress-preserving:** an operator-injected sidecar in the driver pod accepts small progress markers from the application, commits them to the store under the current epoch, and restores the latest marker into the restarted driver, so a well-structured job resumes from its last commit point instead of from zero.

This is **not** driver checkpoint/restore — Spark has no such thing, and executors/shuffle/scheduler state are lost regardless of what the operator does. It's restart-minus-redone-work plus a correctness guarantee that the previous driver can never double-commit.

### Why is this needed?

The driver is the SPoF of Spark on Kubernetes. Today `restartPolicy: OnFailure/Always` has two gaps for long-running batch/iterative-ML jobs, especially on spot/preemptible capacity:

- **Correctness:** after a node partition, a rerun can race a still-running zombie of the old driver on non-transactional sinks (plain files, JDBC, custom sinks) — nothing fences the old driver at submission time. (Transactional sinks like Delta/Iceberg already protect their own commits.)
- **Economics:** the rerun starts from zero even when the job had durably completed most of its work, because the operator gives applications no standard place to record and read back progress.

YARN has a precedent for restart-with-attempt-tracking living in the resource-manager layer (`spark.yarn.maxAppAttempts`). On Kubernetes, the Spark Operator is that layer.

### Describe the solution you would like

Reuse machinery the operator already has, rather than a new CRD or a second controller:

- The existing restart state machine (`FAILING → PENDING_RERUN → SUBMITTED`) gains one step: atomically advance the fencing epoch before resubmission. Correctness never depends on the old driver actually being dead — only on it being fenced. Order: **fence → delete → resubmit**, and resubmission stays the native `spark-submit` path.
- The existing driver pod mutating webhook injects a `spark-recovery-agent` sidecar that heartbeats to the store, serves a localhost progress-marker API to the driver, and restores the latest committed marker before the driver boots.
- `status.recoveryStatus` reports the current epoch and which epoch's marker (if any) was restored. A warning event fires if a recovery-enabled application restarts having never committed a marker, so a fenced restart-from-zero never silently masquerades as progress preservation.
- Entirely opt-in via `spec.restartPolicy.recovery`, gated behind a new `FencedRestart` alpha feature flag; zero behavior change when unset.

I have a working implementation ready (design doc, API types, Redis-backed state store, webhook sidecar injection, controller integration, docs, and an end-to-end kind-based demo that kills a driver mid-run and verifies fenced resume) and would like to open a PR against this issue.

### Describe alternatives you have considered

- **External companion operator/sidecar-only add-on:** can't compose with the operator's own restart state machine, so it either races it or has to reimplement large parts of it.
- **CRIU / process checkpointing:** doesn't fit Spark's distributed executor/shuffle model — checkpointing the driver process alone doesn't recover executors or shuffle state, and is a much larger surface area for a much smaller correctness win.
- **Lease-based fencing only (no progress preservation):** solves the correctness half but leaves the economics problem (restart-from-zero) unaddressed, which is the more common pain point for long jobs.
- **Sink-side transactions only (rely on Delta/Iceberg etc.):** doesn't help plain-file, JDBC, or custom sinks, and doesn't address restart-from-zero either.

### Additional context

Design doc / KEP-style proposal covering goals, non-goals, the fencing protocol, and graduation criteria will be included in the PR at `proposals/fenced-progress-preserving-restart.md`.

Contributor guide

Open the contributing guide

Research direction

Read proposals/fenced-progress-preserving-restart.md first, then trace the existing restart state machine and driver pod mutating webhook mentioned in the issue. Review how restartPolicy and recoveryStatus are represented, along with the kind-based end-to-end demo. Done means the opt-in feature supports fencing and progress restoration without changing behavior when unset, with the described status and warning event verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, redis
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.