[SUPPORT] Query with MDT but time cost is much as not on mdt
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. Data write to Hudi table and mdt is open;
2.(set hoodie.metadata.enable=true;
set hoodie.enable.data.skipping=true;
set hoodie.metadata.index.column.stats.enable=true;)
3. Query data with cols filter
4. Data query is slow as before use mdt
**Expected behavior**
**If query on ro table it cost 5min,but query rt with mdt 20min more time would cost.**
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.2.0
* Hive version : 1.1.0
* Hadoop version : 3.2
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is named. Start by reproducing the comparison between read-optimized and real-time queries with Hudi 0.14.0, Spark 3.2.0, Hadoop 3.2, HDFS, and the three metadata settings shown. Done requires isolating why the metadata-enabled query takes longer and documenting a reproducible result or specific fix scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100