apache / apache/hudi

Hive Sync Fails with TTransportException When Using Kerberos Authentication

Open
#18,077 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

Title: Hive Sync Fails with TTransportException When Using Kerberos Authentication

Environment:

Hudi: 1.1.1

Flink: 1.20.3

Hive: 3.1.2

Authentication: Kerberos enabled

Problem Description:
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to create HiveMetaStoreClient
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:112) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:279) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4606) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:4357) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:4345) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2861) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_301]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_301]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_301]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_301]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_301]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at com.sun.proxy.$Proxy72.getAllFunctions(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:435) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:45) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:92) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:137) ~[hudi-flink1.20-bundle-1.1.1.jar:1.1.1]
... 8 more

Key Log Errors:

set_ugi() not successful, Likely cause: new client talking to old server.

MetaStoreClient lost connection with TTransportException

Failed to register all functions due to TTransportException

Final error: Failed to create HiveMetaStoreClient → Got runtime exception when hive syncing

Hudi Table Configuration (relevant sync settings):

sql
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.kerberos.enable' = 'true',
'hive_sync.kerberos.principal' = '...',
'hive_sync.metastore.uris' = 'thrift://100.150.0.23:9083',
'hive_sync.db' = 'jxk',
'hive_sync.table' = 'c_bton_cdc100222223hudi2'
What Works:

Hudi data writes and commits succeed (Commit instant success!)

Compaction scheduling works normally

flink/lib
flink-cep-1.20.3.jar
flink-table-api-java-uber-1.20.3.jar
flink-scala_2.12-1.20.3.jar
flink-connector-files-1.20.3.jar
flink-table-runtime-1.20.3.jar
log4j-core-2.24.3.jar
log4j-slf4j-impl-2.24.3.jar
log4j-1.2-api-2.24.3.jar
flink-json-1.20.3.jar
flink-table-planner-loader-1.20.3.jar
log4j-api-2.24.3.jar
flink-csv-1.20.3.jar
mysql-connector-j-8.0.33.jar
flink-connector-jdbc-3.3.0-1.20.jar
flink-dist-1.20.3.jar
flink-cdc-pipeline-connector-doris-3.5.0.jar
flink-cdc-pipeline-connector-mysql-3.5.0.jar
flink_udf-1.0-SNAPSHOT.jar
flink-cdc-pipeline-connector-postgres-3.5.0.jar
flink-doris-connector-1.20-25.1.0.jar
ojdbc8-21.1.0.0.jar
flink-sql-connector-oracle-cdc-3.5.0.jar
htrace-core-3.1.0-incubating.jar
seatunnel-core-flink.jar
hudi-flink1.20-bundle-1.1.1.jar
guava-27.0.1-jre.jar
hadoop-mapreduce-client-core-3.2.2.jar
calcite-core-1.10.0.jar

What Fails:

Hive metadata synchronization step fails consistently

Hive Metastore connection/client initialization fails with transport exceptions

Additional Context:
The Hive Metastore server is Kerberos-enabled (principal: hive/_HOST@SUGON.COM). The client appears to connect initially but fails during subsequent operations like getAllFunctions.

Question:
Is there a known compatibility issue between Hudi 1.1.1 (Flink bundle) and Hive 3.1.2 with Kerberos authentication, particularly regarding Thrift transport? Are there specific configuration requirements or workarounds for this setup?

Let me know if you'd like to include any additional details or adjust the tone.

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace Hive sync initialization through HoodieHiveSyncClient.java, HiveSyncTool.java, and IMetaStoreClientUtil.java, focusing on the getAllFunctions call shown in the stack trace. Reproduce the Kerberos-enabled Hive sync with the listed settings and determine whether the failure is a compatibility or configuration issue; done means identifying a confirmed workaround or a narrowly scoped fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.