[SparkConnect] Support url path in SparkConnect connection string
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
Current Spark Connect uses connection string like `sc://host1:15002`, which does not support a path in the url, e.g. not support value like `sc://host1/path1:15002`.
In Kubernetes, people can set up an Ingress to expose Spark Connect driver endpoint by a matching url, e.g. `http://host1/path1` can be routed to the driver endpoint. Need to update Spark Connect client to support this scenario.
This could be a Spark Client side change, e.g. in Python Spark Connect client, connect to server based on the host name, and intercept GPRC call to avoid path interfering GRPC protocol.
I am working on a prototype, also open to hear feedback / suggestions.
Contributor guide
Research direction
Start with the Spark Connect client and the prototype described in the issue, focusing first on connection-string parsing and how the client establishes gRPC calls. Confirm that a URL path can route through a Kubernetes Ingress without interfering with the gRPC protocol; done means Spark Connect connections using a path such as sc://host1/path1:15002 work successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, kubernetes, python
- Domain
- backend-api-design, distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100