Hive Sync Fails with TTransportException When Using Kerberos Authentication
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
Title: Hive Sync Fails with TTransportException When Using Kerberos Authentication
Environment:
Hudi: 1.1.1
Flink: 1.20.3
Hive: 3.1.2
Authentication: Kerberos enabled
Problem Description:
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to create HiveMetaStoreClient
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:112) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:279) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4606) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:4357) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:4345) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2861) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_301]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_301]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Key Log Errors:
set_ugi() not successful, Likely cause: new client talking to old server.
MetaStoreClient lost connection with TTransportException
Failed to register all functions due to TTransportException
Final error: Failed to create HiveMetaStoreClient → Got runtime exception when hive syncing
Hudi Table Configuration (relevant sync settings):
sql
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.kerberos.enable' = 'true',
'hive_sync.kerberos.principal' = '...',
'hive_sync.metastore.uris' = 'thrift://100.150.0.23:9083',
'hive_sync.db' = 'jxk',
'hive_sync.table' = 'c_bton_cdc100222223hudi2'
What Works:
Hudi data writes and commits succeed (Commit instant success!)
Compaction scheduling works normally
flink/lib
flink-cep-1.20.3.jar
flink-table-api-java-uber-1.20.3.jar
flink-scala_2.12-1.20.3.jar
flink-connector-files-1.20.3.jar
flink-table-runtime-1.20.3.jar
log4j-core-2.24.3.jar
log4j-slf4j-impl-2.24.3.jar
log4j-1.2-api-2.24.3.jar
flink-json-1.20.3.jar
flink-table-planner-loader-1.20.3.jar
log4j-api-2.24.3.jar
flink-csv-1.20.3.jar
mysql-connector-j-8.0.33.jar
flink-connector-jdbc-3.3.0-1.20.jar
flink-dist-1.20.3.jar
flink-cdc-pipeline-connector-doris-3.5.0.jar
flink-cdc-pipeline-connector-mysql-3.5.0.jar
flink_udf-1.0-SNAPSHOT.jar
flink-cdc-pipeline-connector-postgres-3.5.0.jar
flink-doris-connector-1.20-25.1.0.jar
ojdbc8-21.1.0.0.jar
flink-sql-connector-oracle-cdc-3.5.0.jar
htrace-core-3.1.0-incubating.jar
seatunnel-core-flink.jar
hudi-flink1.20-bundle-1.1.1.jar
guava-27.0.1-jre.jar
hadoop-mapreduce-client-core-3.2.2.jar
calcite-core-1.10.0.jar
What Fails:
Hive metadata synchronization step fails consistently
Hive Metastore connection/client initialization fails with transport exceptions
Additional Context:
The Hive Metastore server is Kerberos-enabled (principal: hive/_HOST@SUGON.COM). The client appears to connect initially but fails during subsequent operations like getAllFunctions.
Question:
Is there a known compatibility issue between Hudi 1.1.1 (Flink bundle) and Hive 3.1.2 with Kerberos authentication, particularly regarding Thrift transport? Are there specific configuration requirements or workarounds for this setup?
Let me know if you'd like to include any additional details or adjust the tone.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace Hive sync initialization through HoodieHiveSyncClient.java, HiveSyncTool.java, and IMetaStoreClientUtil.java, focusing on the getAllFunctions call shown in the stack trace. Reproduce the Kerberos-enabled Hive sync with the listed settings and determine whether the failure is a compatibility or configuration issue; done means identifying a confirmed workaround or a narrowly scoped fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100