apache / apache/hudi

[SUPPORT] Query with MDT but time cost is much as not on mdt

Open
#10,819 10 comments 0 reactions 0 assignees View on GitHub
area:index area:metadata-table component:data-skipping priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1. Data write to Hudi table and mdt is open;
2.(set hoodie.metadata.enable=true;
set hoodie.enable.data.skipping=true;
set hoodie.metadata.index.column.stats.enable=true;)
3. Query data with cols filter
4. Data query is slow as before use mdt

**Expected behavior**

**If query on ro table it cost 5min,but query rt with mdt 20min more time would cost.**

**Environment Description**

* Hudi version : 0.14.0

* Spark version : 3.2.0

* Hive version : 1.1.0

* Hadoop version : 3.2

* Storage (HDFS/S3/GCS..) : HDFS

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is named. Start by reproducing the comparison between read-optimized and real-time queries with Hudi 0.14.0, Spark 3.2.0, Hadoop 3.2, HDFS, and the three metadata settings shown. Done requires isolating why the metadata-enabled query takes longer and documenting a reproducible result or specific fix scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.