[SUPPORT] When Doris queries Hudi, an error was reported that the partition metadata file of Hudi cannot be opened
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
A clear and concise description of the problem.
**When Doris queries Hudi, an error was reported that the partition metadata file of Hudi cannot be opened.**
Steps to reproduce the behavior:
1). Creating a hivecatalog in Dorise
CREATE CATALOG hive PROPERTIES (
'type'='hms',
'hive.metastore.uris' = 'thrift://172.0.0.1:9083',
'hive.metastore.sasl.enabled' = 'true',
'hive.metastore.kerberos.principal' = 'your-hms-principal',
'hive.metastore.kerberos.kerberos' = 'your.keytab'
'dfs.nameservices'='your-nameservice',
'dfs.ha.namenodes.your-nameservice'='nn1,nn2',
'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:8088',
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:8088',
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider',
'hadoop.security.authentication' = 'kerberos',
'hadoop.kerberos.keytab' = '/your-keytab-filepath/your.keytab',
'hadoop.kerberos.principal' = 'your-principal@YOUR.COM',
'yarn.resourcemanager.principal' = 'your-rm-principal',
'hive.version' = '1.1.0'
);
2). Execute the following SQL to query the data
select * from hive.hudi.table ;
3). For the same table, execute the following SQL query to report an error.。
select count(*) from hive.hudi.table;
**Expected behavior**
1). Doris logs reported an error as follows


Being able to read parquet files correctly, but unable to read ".hoodie_ Partition_ Metadata" file
**Environment Description**
* Hudi version : 0.12.0
* Doris version : 2.0.0
* Hive version : 2.1.1
* Hadoop version :3.0.0
* Storage (HDFS/S3/GCS..) :HDFS
* Running on Docker? (yes/no) :no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reproduction steps and attached Doris logs, comparing the successful SELECT query with the failing COUNT query. Investigate the Hudi 0.12.0, Doris 2.0.0, Hive 2.1.1, and Hadoop 3.0.0 HDFS integration, and confirm that both Parquet data and the .hoodie partition metadata can be read correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, sql
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100