apache / apache/gravitino

[Improvement] Spark Connector Need DelegationTokenProvider for k8s deployment

Open
#3,297 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

When deploying a spark application on k8s and then connecting multiple Hive Metastore (cluster mode), the spark needs DelegationTokenProvider to get delegate tokens from different HMS in the submitting stage and store them at the UserGroupInformation so that the spark driver can communicate with HMS.

For example, KyuubiHiveConnector contains the KyuubiHiveConnectorDelegationTokenProvider to deal with this case.

Now, the Gravitino Spark Connector depends on the KyuubiHiveConnector. However, the KyuubiHiveConnectorDelegationTokenProvider filters the catalog's implementation, which doesn't work in the above case. In addition, It is only for the hive catalog, not including the iceberg catalog.

The Gravitino Spark Connector needs its DelegationTokenProvider to handle this case and ensure it works well in both hive and iceberg catalogs under a Kerberos environment.

REF:
https://github.com/apache/kyuubi/blob/master/extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/KyuubiHiveConnectorDelegationTokenProvider.scala

https://github.com/apache/kyuubi/pull/4560

### How should we improve?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading KyuubiHiveConnectorDelegationTokenProvider.scala and the referenced Kyuubi pull request to understand the existing delegation-token approach. Then locate the Gravitino Spark Connector entry points and determine how a provider can cover both Hive and Iceberg catalogs in Kerberos-based Kubernetes deployments. Done means the connector obtains and stores tokens for both catalog types so the Spark driver can communicate with their metastore services.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, scala
Domain
data-engineering, distributed-systems, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.