[FEATURE][SERVER] Support Spark Connect protocol — make Kyuubi a multi-tenant gRPC gateway for Spark Connect clients
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Spark Connect (GA in Apache Spark 3.4) defines a language-agnostic gRPC protocol that decouples clients from the Spark driver. Today Kyuubi only exposes Thrift/JDBC/REST surfaces; PySpark users connecting via `spark.remote("sc://...")` or any non-JVM thin client (Go, Rust) cannot benefit from Kyuubi's multi-tenancy, session pooling, and access control.
Supporting Spark Connect transforms Kyuubi into a unified gateway for both legacy JDBC workloads and modern DataFrame/Spark Connect clients, with no changes required on the Spark engine side.
## Motivation
| Today | With this feature |
|---|---|
| PySpark thin clients bypass Kyuubi entirely | All Spark Connect clients authenticate through Kyuubi |
| Non-JVM clients (Go, Rust) have no path to Kyuubi | Any Spark Connect-compatible client works |
| JDBC is the only managed multi-tenant surface | gRPC + JDBC both benefit from Kyuubi's session pooling and access control |
## Proposed Architecture
```mermaid
flowchart TD
subgraph Clients
A1["PySpark\nspark.remote('sc://...')"]
A2["Go / Rust\nSpark Connect client"]
end
subgraph KyuubiServer ["Kyuubi Server (kyuubi-spark-connect module)"]
B["NettyGrpcServer\n:15002"]
C["AuthInterceptor\n(Bearer token → KyuubiUser)"]
D["SparkConnectFrontendService\n(SparkConnectServiceGrpc)"]
E["SparkConnectSessionManager\n(session lifecycle, user isolation)"]
F["SparkConnectEngineProxy\n(gRPC channel to engine)"]
G["ServiceDiscovery\n(ZooKeeper / Kubernetes)"]
end
subgraph EngineAlice ["Spark Engine – user alice"]
H1["SparkConnect gRPC\n(embedded, auto-port)"]
H2["SparkSession"]
end
subgraph EngineBob ["Spark Engine – user bob"]
I1["SparkConnect gRPC\n(embedded, auto-port)"]
I2["SparkSession"]
end
A1 -- "gRPC (Spark Connect proto)" --> B
A2 -- "gRPC (Spark Connect proto)" --> B
B --> C
C -- "KyuubiUser" --> D
D --> E
E -- "open / reuse session" --> F
F -- "discover sparkConnectPort" --> G
F -- "proxy protobuf stream\n(alice)" --> H1
F -- "proxy protobuf stream\n(bob)" --> I1
H1 --- H2
I1 --- I2
```
**Key design choice — proxy, not embed:** Kyuubi forwards Spark Connect protobuf streams to a Spark engine running in `--spark-connect` mode. Kyuubi does not embed a SparkSession, preserving the fundamental server/engine module boundary.
## Proposed Changes
- **New module** `kyuubi-spark-connect` (profile-gated via `-Pspark-connect`) containing the gRPC server, protobuf stubs, and session bridge.
- **New gRPC listener** implementing `SparkConnectService` proto on port 15002 (configurable), alongside the existing Thrift frontend.
- **Session management extension**: Spark Connect sessions map onto Kyuubi's existing `SessionManager` / `OperationManager` with a new `SparkConnectSession` type; per-user engine limits and idle-timeout apply unchanged.
- **Engine-side proxy**: transparent byte-level forwarding of `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, and `ReattachExecute` RPCs to the per-user Spark engine.
- **Engine port discovery**: engine registers its Spark Connect gRPC port in ZooKeeper (node attribute `sparkConnectPort`) or Kubernetes (pod label `kyuubi.apache.org/spark-connect-port`).
- **Auth**: `AuthInterceptor` validates bearer tokens via Kyuubi's existing `AuthenticationProvider` chain; server-authenticated identity overrides any client-supplied `UserContext.user_name`.
- **New config namespace** `kyuubi.frontend.spark.connect.*` (enabled, bind host/port, max message size, TLS options).
- Feature off by default (`kyuubi.frontend.spark.connect.enabled=false`); fully additive and non-breaking.
## New Dependencies
| Artifact | License | Scope |
|---|---|---|
| `grpc-netty-shaded` | Apache 2.0 | `-Pspark-connect` only |
| `protobuf-java` | BSD-3 | `-Pspark-connect` only |
| `spark-connect-common` (Spark 3.5+) | Apache 2.0 | `-Pspark-connect` only |
## Open Questions for Community Discussion
1. Should the initial scope include `ReattachExecute` (required for long-running streaming queries), or defer to a follow-up?
2. Should `AddArtifacts` (client-streaming file/JAR upload) be in scope for v1?
3. Should `kyuubi-spark-connect` live as a top-level module or under `extensions/server/`?
4. How should the Spark Connect auth token model interact with Ranger/HMS auth plugins — does the bearer token carry the same identity used for SQL authorization?
5. Which Spark Connect proto version to target first? (Spark 3.5 is the most widely deployed LTS, but Spark 4.0 changed the proto significantly.)
## Related
- Planning artifacts (proposal, design, specs, tasks): #7510
- Apache Spark Connect documentation: https://spark.apache.org/docs/latest/spark-connect-overview.html
- `SparkConnectService` proto: `connector/connect/common/src/main/protobuf/spark/connect/base.proto` in the Spark repo
Contributor guide
Research direction
Start with planning issue #7510, then inspect Kyuubi's existing SessionManager, OperationManager, AuthenticationProvider chain, and the Spark Connect proto at connector/connect/common/src/main/protobuf/spark/connect/base.proto. Trace how the proposed kyuubi-spark-connect module would connect the gRPC listener, session lifecycle, engine port discovery, and proxy RPCs. Done requires an agreed scope for the open protocol, module, authentication, and deployment questions before implementation begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, kubernetes, scala
- Domain
- api, authentication, backend, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100