apache / apache/kyuubi

[TASK][CHALLENGE] Support Spark Connect Frontend/Backend

Open
#5,383 26 comments 6 reactions 1 assignee Claimed by @davidyuan1223 View on GitHub
hacktoberfest
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before creating

- [X] I have searched in the [task list](https://github.com/orgs/apache/projects/296) and found no similar tasks.

### Mentor

- [X] I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

### Skill requirements

- Knowledge about Spark Connect
- Knowledge about Kyuubi architecture
- Knowledge about protobuf
- Knowledge about grpc
- Knowledge about thrift

### Background and Goals

Make Kyuubi server compatible with Spark Connect protocol, so that people can use Spark Connect client to connect to Kyuubi Server.

image

### Implementation steps

1. Add a new Spark Connect frontend
1.1 Add basic gRpc server as frontend
1.2 Compatible with Spark Connect protocol, see https://github.com/apache/spark/blob/master/connector/connect/common/src/main/protobuf/spark/connect/base.proto
1.3 Support ExecutePlan
1.4 Support AnalyzePlan
1.5 Support Config
1.6 Support AddArtifacts
1.7 Support ArtifactsStatus
1.8 Support Interrupt
1.9 Support ReattachExecute
1.10 Support ReleaseExecute
1.11 Serialize the protobuf based request

2. Add a new Spark Connect backend
2.1 Imprort Sprak-Connect-Server and rewrite SparkConnectService https://github.com/apache/spark/blob/master/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala
2.2 Deserialize response to protobuf based

3. Add IT
4. Add docs

### Additional context

Introduction of https://github.com/apache/kyuubi/issues/6232

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.