kubeflow / kubeflow/spark-operator

[FEATURE] REST-based Spark submitter service as pluggable alternative to spark-submit CLI

Open
#2,972 13 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What feature you would like to be added?

Add a REST-based Spark submitter service as a pluggable alternative to the default `spark-submit` CLI invocation, implementing the existing `SparkApplicationSubmitter` interface.

### Why is this needed?

The current `spark-submit` approach spawns a full JVM process per submission, causing:

1. **JVM spin-up overhead** — Each submission pays ~2-8s of JVM startup cost, which compounds under load and is the primary bottleneck for operators handling 1000s+ of SparkApplications (as documented in #2337)
2. **Resource waste** — Each `spark-submit` process consumes ~200-400MB of memory that competes with the operator's own memory budget
3. **Poor scalability** — Degrades rapidly under high submission rates due to per-submission JVM overhead

### Describe the solution you would like

Two new components:

1. **`RESTSparkApplicationSubmitter`** (Go, in the operator) — A REST client that implements the `SparkApplicationSubmitter` interface. Instead of spawning `spark-submit` via `exec.Command`, it POSTs the spark-submit args to the submitter service over HTTP.

2. **Spark Submitter Service** (Scala) — A new long-running REST service deployed as its own K8s Deployment (similar to the existing controller and webhook deployments). It receives spark-submit args, uses Spark's internal libraries to build the driver spec, and creates the driver pod directly via the K8s API.

#### How it works

1. The operator's existing `buildSparkSubmitArgs()` converts SparkApplication CRD → spark-submit CLI args
2. `RESTSparkApplicationSubmitter` POSTs those args to the submitter service
3. The submitter service uses Spark's internal libraries (`SparkSubmitArguments`, `KubernetesDriverBuilder`) to create the driver pod directly — no JVM process spawn per submission

#### Integration contract (REST API)

```
POST /spark-submit
Content-Type: application/json

{
"spark_submit_args": ["--master", "k8s://...", "--deploy-mode", "cluster", ...]
}
```

Response (201 Created):
```json
{
"driver_pod_name": "my-app-abc123-driver",
"namespace": "spark-jobs",
"spark_app_id": "spark-b992db7da52c42298736dcbb3c9142be"
}
```

#### Why spark-submit args as the contract?

- **Zero spec translation** — the operator already has `buildSparkSubmitArgs()` that converts SparkApplication CRD to CLI args. No new schema to maintain.
- **Full Spark compatibility** — the service uses Spark's internal `SparkSubmitArguments` and `SparkSubmit.prepareSubmitEnvironment`, so every `--conf` option that works with `spark-submit` works here identically.
- **Thin integration** — the operator-side change is ~100 lines (HTTP POST instead of `exec.Command`).

#### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│ Spark Operator Controller │
│ │
│ SparkApplication CR ──▶ buildSparkSubmitArgs() ──▶ HTTP POST │
│ (existing code) │
│ │
│ (RESTSparkApplicationSubmitter - REST client) │
└─────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────┐
│ Spark Submitter Service (K8s Deployment - Scala) │
│ │
│ spark-submit args ──▶ SparkSubmitArguments ──▶ KubernetesDriver │
│ (Spark internal) Builder │
│ │ │
│ ▼ │
│ K8s API (fabric8) │
│ Create driver pod │
└─────────────────────────────────────────────────────────────────┘
```

#### Key characteristics of the submitter service

- **Zero additional dependencies** — The service JAR (~1MB) requires no extra Java/Scala dependency JARs beyond what's already in the standard Spark base image. All Spark, Kubernetes client, and Jackson libraries are provided by the base image at runtime.
- **~200ms submission latency** (vs 2-8s with spark-submit CLI)
- **Long-running JVM** — amortizes startup cost across all submissions
- **Fire-and-forget** — returns immediately after driver pod creation
- **Spark-native parsing** — uses `SparkSubmit.prepareSubmitEnvironment` for full compatibility with all Spark conf options
- **Fixes ConfigMap singleton issue** — Spark's built-in K8s client uses a static ConfigMap name, causing collisions in multi-submission JVMs; the service generates unique names
- **Supports Java, Python, and R** applications
- **Pod template support** — accepts driver/executor pod templates as nested JSON

#### Changes to the operator

1. Add `RESTSparkApplicationSubmitter` struct implementing `SparkApplicationSubmitter` interface (REST client that POSTs args to the submitter service)
2. Add Helm values for configuring the submitter mode (`spark-submit` | `rest-service`)
3. Add K8s Deployment manifest for the submitter service (alongside existing controller and webhook deployments)
4. The operator's existing `buildSparkSubmitArgs()` function is reused as-is

### Describe alternatives you have considered

1. **Native Go reimplementation (#2337)** — Requires reimplementing Spark's argument resolution, driver spec building, and K8s resource creation in Go. High effort, hard to keep in sync with Spark releases.
2. **Keep current spark-submit** — Workable at low scale but degrades rapidly under high submission rates due to per-submission JVM overhead.

### Additional context

- The submitter service runs on top of the **standard Spark base image** with only a thin ~1MB service JAR added — no additional dependency JARs needed
- Tested with Spark 4.0.1 on Kubernetes
- Related issues: #2337

### Willingness to contribute

I am willing to contribute this feature, including:
- The `RESTSparkApplicationSubmitter` implementation in the operator (Go)
- Helm chart changes and K8s Deployment for the submitter service
- Documentation
- The submitter service itself (already implemented and tested)

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start with the existing SparkApplicationSubmitter interface and buildSparkSubmitArgs() entry point in the operator, then inspect the current controller and webhook Helm deployment manifests. Review the proposed REST contract and Spark submitter service design before splitting the work. Done means the REST submitter, configuration, deployment, and service integrate with existing submissions and have tests or validation for the documented response.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, scala, spark
Domain
api, backend, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.