[Improvement] Spark Connector Need DelegationTokenProvider for k8s deployment
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
When deploying a spark application on k8s and then connecting multiple Hive Metastore (cluster mode), the spark needs DelegationTokenProvider to get delegate tokens from different HMS in the submitting stage and store them at the UserGroupInformation so that the spark driver can communicate with HMS.
For example, KyuubiHiveConnector contains the KyuubiHiveConnectorDelegationTokenProvider to deal with this case.
Now, the Gravitino Spark Connector depends on the KyuubiHiveConnector. However, the KyuubiHiveConnectorDelegationTokenProvider filters the catalog's implementation, which doesn't work in the above case. In addition, It is only for the hive catalog, not including the iceberg catalog.
The Gravitino Spark Connector needs its DelegationTokenProvider to handle this case and ensure it works well in both hive and iceberg catalogs under a Kerberos environment.
REF:
https://github.com/apache/kyuubi/blob/master/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/KyuubiHiveConnectorDelegationTokenProvider.scala
https://github.com/apache/kyuubi/pull/4560
### How should we improve?
_No response_
Contributor guide
Research direction
Start by reading KyuubiHiveConnectorDelegationTokenProvider.scala and the referenced Kyuubi pull request to understand the existing delegation-token approach. Then locate the Gravitino Spark Connector entry points and determine how a provider can cover both Hive and Iceberg catalogs in Kerberos-based Kubernetes deployments. Done means the connector obtains and stores tokens for both catalog types so the Spark driver can communicate with their metastore services.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, scala
- Domain
- data-engineering, distributed-systems, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100