apache / apache/hudi

[SUPPORT] When Doris queries Hudi, an error was reported that the partition metadata file of Hudi cannot be opened

Open
#9,568 2 comments 0 reactions 0 assignees View on GitHub
area:query-engine priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

A clear and concise description of the problem.

**When Doris queries Hudi, an error was reported that the partition metadata file of Hudi cannot be opened.**

Steps to reproduce the behavior:

1). Creating a hivecatalog in Dorise
CREATE CATALOG hive PROPERTIES (
'type'='hms',
'hive.metastore.uris' = 'thrift://172.0.0.1:9083',
'hive.metastore.sasl.enabled' = 'true',
'hive.metastore.kerberos.principal' = 'your-hms-principal',
'hive.metastore.kerberos.kerberos' = 'your.keytab'
'dfs.nameservices'='your-nameservice',
'dfs.ha.namenodes.your-nameservice'='nn1,nn2',
'dfs.namenode.rpc-address.your-nameservice.nn1'='172.21.0.2:8088',
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:8088',
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider',
'hadoop.security.authentication' = 'kerberos',
'hadoop.kerberos.keytab' = '/your-keytab-filepath/your.keytab',
'hadoop.kerberos.principal' = 'your-principal@YOUR.COM',
'yarn.resourcemanager.principal' = 'your-rm-principal',
'hive.version' = '1.1.0'
);
2). Execute the following SQL to query the data
select * from hive.hudi.table ;

3). For the same table, execute the following SQL query to report an error.。
select count(*) from hive.hudi.table;

**Expected behavior**
1). Doris logs reported an error as follows
![image](https://github.com/apache/hudi/assets/58579429/70789407-f1f5-4726-9183-d7ff4252de86)

![image](https://github.com/apache/hudi/assets/58579429/c9ebd0c5-a374-402d-89d0-00772d898869)

Being able to read parquet files correctly, but unable to read ".hoodie_ Partition_ Metadata" file

**Environment Description**

* Hudi version : 0.12.0

* Doris version : 2.0.0

* Hive version : 2.1.1

* Hadoop version :3.0.0

* Storage (HDFS/S3/GCS..) :HDFS

* Running on Docker? (yes/no) :no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproduction steps and attached Doris logs, comparing the successful SELECT query with the failing COUNT query. Investigate the Hudi 0.12.0, Doris 2.0.0, Hive 2.1.1, and Hadoop 3.0.0 HDFS integration, and confirm that both Parquet data and the .hoodie partition metadata can be read correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, sql
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.