apache / apache/kyuubi

[FEATURE][SERVER] Support Spark Connect protocol — make Kyuubi a multi-tenant gRPC gateway for Spark Connect clients

Open
#7,511 10 comments 5 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

## Summary

Spark Connect (GA in Apache Spark 3.4) defines a language-agnostic gRPC protocol that decouples clients from the Spark driver. Today Kyuubi only exposes Thrift/JDBC/REST surfaces; PySpark users connecting via `spark.remote("sc://...")` or any non-JVM thin client (Go, Rust) cannot benefit from Kyuubi's multi-tenancy, session pooling, and access control.

Supporting Spark Connect transforms Kyuubi into a unified gateway for both legacy JDBC workloads and modern DataFrame/Spark Connect clients, with no changes required on the Spark engine side.

## Motivation

| Today | With this feature |
|---|---|
| PySpark thin clients bypass Kyuubi entirely | All Spark Connect clients authenticate through Kyuubi |
| Non-JVM clients (Go, Rust) have no path to Kyuubi | Any Spark Connect-compatible client works |
| JDBC is the only managed multi-tenant surface | gRPC + JDBC both benefit from Kyuubi's session pooling and access control |

## Proposed Architecture

```mermaid
flowchart TD
subgraph Clients
A1["PySpark\nspark.remote('sc://...')"]
A2["Go / Rust\nSpark Connect client"]
end

subgraph KyuubiServer ["Kyuubi Server (kyuubi-spark-connect module)"]
B["NettyGrpcServer\n:15002"]
C["AuthInterceptor\n(Bearer token → KyuubiUser)"]
D["SparkConnectFrontendService\n(SparkConnectServiceGrpc)"]
E["SparkConnectSessionManager\n(session lifecycle, user isolation)"]
F["SparkConnectEngineProxy\n(gRPC channel to engine)"]
G["ServiceDiscovery\n(ZooKeeper / Kubernetes)"]
end

subgraph EngineAlice ["Spark Engine – user alice"]
H1["SparkConnect gRPC\n(embedded, auto-port)"]
H2["SparkSession"]
end

subgraph EngineBob ["Spark Engine – user bob"]
I1["SparkConnect gRPC\n(embedded, auto-port)"]
I2["SparkSession"]
end

A1 -- "gRPC (Spark Connect proto)" --> B
A2 -- "gRPC (Spark Connect proto)" --> B
B --> C
C -- "KyuubiUser" --> D
D --> E
E -- "open / reuse session" --> F
F -- "discover sparkConnectPort" --> G
F -- "proxy protobuf stream\n(alice)" --> H1
F -- "proxy protobuf stream\n(bob)" --> I1
H1 --- H2
I1 --- I2
```

**Key design choice — proxy, not embed:** Kyuubi forwards Spark Connect protobuf streams to a Spark engine running in `--spark-connect` mode. Kyuubi does not embed a SparkSession, preserving the fundamental server/engine module boundary.

## Proposed Changes

- **New module** `kyuubi-spark-connect` (profile-gated via `-Pspark-connect`) containing the gRPC server, protobuf stubs, and session bridge.
- **New gRPC listener** implementing `SparkConnectService` proto on port 15002 (configurable), alongside the existing Thrift frontend.
- **Session management extension**: Spark Connect sessions map onto Kyuubi's existing `SessionManager` / `OperationManager` with a new `SparkConnectSession` type; per-user engine limits and idle-timeout apply unchanged.
- **Engine-side proxy**: transparent byte-level forwarding of `ExecutePlan`, `AnalyzePlan`, `Config`, `Interrupt`, and `ReattachExecute` RPCs to the per-user Spark engine.
- **Engine port discovery**: engine registers its Spark Connect gRPC port in ZooKeeper (node attribute `sparkConnectPort`) or Kubernetes (pod label `kyuubi.apache.org/spark-connect-port`).
- **Auth**: `AuthInterceptor` validates bearer tokens via Kyuubi's existing `AuthenticationProvider` chain; server-authenticated identity overrides any client-supplied `UserContext.user_name`.
- **New config namespace** `kyuubi.frontend.spark.connect.*` (enabled, bind host/port, max message size, TLS options).
- Feature off by default (`kyuubi.frontend.spark.connect.enabled=false`); fully additive and non-breaking.

## New Dependencies

| Artifact | License | Scope |
|---|---|---|
| `grpc-netty-shaded` | Apache 2.0 | `-Pspark-connect` only |
| `protobuf-java` | BSD-3 | `-Pspark-connect` only |
| `spark-connect-common` (Spark 3.5+) | Apache 2.0 | `-Pspark-connect` only |

## Open Questions for Community Discussion

1. Should the initial scope include `ReattachExecute` (required for long-running streaming queries), or defer to a follow-up?
2. Should `AddArtifacts` (client-streaming file/JAR upload) be in scope for v1?
3. Should `kyuubi-spark-connect` live as a top-level module or under `extensions/server/`?
4. How should the Spark Connect auth token model interact with Ranger/HMS auth plugins — does the bearer token carry the same identity used for SQL authorization?
5. Which Spark Connect proto version to target first? (Spark 3.5 is the most widely deployed LTS, but Spark 4.0 changed the proto significantly.)

## Related

- Planning artifacts (proposal, design, specs, tasks): #7510
- Apache Spark Connect documentation: https://spark.apache.org/docs/latest/spark-connect-overview.html
- `SparkConnectService` proto: `connector/connect/common/src/main/protobuf/spark/connect/base.proto` in the Spark repo

Contributor guide

Open the contributing guide

Research direction

Start with planning issue #7510, then inspect Kyuubi's existing SessionManager, OperationManager, AuthenticationProvider chain, and the Spark Connect proto at connector/connect/common/src/main/protobuf/spark/connect/base.proto. Trace how the proposed kyuubi-spark-connect module would connect the gRPC listener, session lifecycle, engine port discovery, and proxy RPCs. Done requires an agreed scope for the open protocol, module, authentication, and deployment questions before implementation begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, kubernetes, scala
Domain
api, authentication, backend, distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.