[Bug]: Kerberos authentication failure
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 395
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 33
Description
### What happened?
When using the External Catalog type, Mixed-Iceberg, Mixed-Hive, Iceberg table formats, and Kerberos authentication method, after creating a hive_catalog with Hive service tickets and keytab files, if you create an Iceberg table in that Catalog and a specific database and insert data, subsequent queries will prompt a client authentication failure. It will look like this:
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:754)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:709)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
at org.apache.hadoop.ipc.Client.call(Client.java:1473)
... 44 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 47 more
After I restarted the Amoro service using ams.sh restart, querying the iceberg table data worked normally. However, if I continued to click the query button, it would throw the authentication exception mentioned above again.
Interestingly, every time I restarted the service, the first query would succeed, but the N query would fail; then after restarting the service again, the first query would succeed, and the N query would fail...
### Affects Versions
0.7.1
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
_No response_
### Relevant log output
_No response_
### Anything else
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start with the AMS query path and the ams.sh restart behavior described in the report, using the Kerberos-authenticated Hive catalog and Iceberg table scenario. Reproduce the first-query-successful, later-query-failing pattern and inspect the available AMS or Hadoop authentication logs. Done means repeated queries no longer fail with the client authentication exception after service startup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- authentication, backend-api-design, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100