apache / apache/kyuubi

[Bug] Connecting to Hive metastore using Kerberos

Open
#5,622 5 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

I have encountered a bug in the Kyuubi server that is causing an undesired behavior. The Kyuubi server is kerberized, meaning that kerberos authentication is required when connecting to it, and as a result, there is a Service Principal Name (SPN) associated with it. The Kyuubi server is started by a user named 'A' and it has a Ticket-Granting Ticket (TGT) cache associated with it. Additionally, the Hive metastore is also kerberized, requiring the spark driver process to communicate with it using Kerberos authentication exclusively.

The issue arises when the spark driver process is spawned/started by the Kyuubi server with the same properties/environment variable (KRB5CCNAME) as the Kyuubi server itself. Since the Kyuubi server periodically runs kinit, the TGT cache gets overwritten with the SPN principal name. As a result, the spark driver process ends up using the SPN principal to connect to the Hive metastore instead of the desired behavior of using the user A principal. Finally, Kyuubi engine throws an error while connecting to Hive metastore saying that no user exist (as Hive metastore is expecting A's principal name but instead it gets SPN)

Here, Kyuubi acts both as a client and server and in these cases, there are two principals involved: one for the server side (acceptor - SPN) and one for when Kyuubi acts as a client (initiator). However, Kyuubi only supports one principal and forces its use on both sides.

### Affects Version(s)

master

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

```yaml
hadoop.security.authentication kerberos
kyuubi.authentication KERBEROS
kyuubi.kinit.principal HTTP/SPN@HOST.COM
kyuubi.kinit.keytab
hadoop.security.auth_to_local.mechanism MIT
kyuubi.authentication.sasl.qop auth-conf
kyuubi.frontend.ssl.keystore.password
kyuubi.frontend.ssl.keystore.path
kyuubi.frontend.ssl.keystore.type JKS
kyuubi.frontend.thrift.binary.ssl.enabled true
```

### Kyuubi Engine Configurations

```yaml
spark.sql.warehouse.dir
spark.worker.cleanup.enabled True
spark.driver.defaultJavaOptions -Djava.security.debug=all -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.jgss.debug=true -Dsun.security.krb5.debug=true -Dsun.security.jgss.native=true -Dsun.security.jgss.lib=/usr/libexec/libgsswrap.so
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:sqlserver://;databaseName=;authenticationScheme=JavaKerberos;trustStore=;encrypt=false;integratedSecurity=true
spark.hadoop.javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver
spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalogImplementation hive
spark.authenticate true
spark.authenticate.secret
spark.network.crypto.enabled true
spark.io.encryption.enabled true
```

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the Kyuubi server runs kinit and how its KRB5CCNAME and principal settings are propagated when a Spark driver is launched. Reproduce the failure with the supplied server and engine configurations, then verify that server authentication still uses the SPN while the Hive metastore connection uses user A's principal.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark
Domain
authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.