apache / apache/spark

[SparkConnect] Support url path in SparkConnect connection string

Open
#56,816 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

Current Spark Connect uses connection string like `sc://host1:15002`, which does not support a path in the url, e.g. not support value like `sc://host1/path1:15002`.

In Kubernetes, people can set up an Ingress to expose Spark Connect driver endpoint by a matching url, e.g. `http://host1/path1` can be routed to the driver endpoint. Need to update Spark Connect client to support this scenario.

This could be a Spark Client side change, e.g. in Python Spark Connect client, connect to server based on the host name, and intercept GPRC call to avoid path interfering GRPC protocol.

I am working on a prototype, also open to hear feedback / suggestions.

Contributor guide

Open the contributing guide

Research direction

Start with the Spark Connect client and the prototype described in the issue, focusing first on connection-string parsing and how the client establishes gRPC calls. Confirm that a URL path can route through a Kubernetes Ingress without interfering with the gRPC protocol; done means Spark Connect connections using a path such as sc://host1/path1:15002 work successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, kubernetes, python
Domain
backend-api-design, distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.