apache / apache/hudi

[SUPPORT] PrestoDB encountered data quality issues while reading the Hudi Mor table.

Open
#8,754 13 comments 0 reactions 1 assignee Claimed by @codope View on GitHub
area:query-engine component:presto component:trino priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
When I used Presto 0.280 (the latest version) to execute SQL statements to query the rows number of the Hudi Mor table, I found a strange phenomenon: sometimes the data rows number is 300w+, and sometimes it becomes 200w+ ,which is abnormal. Please refer to the screenshot below for details.
I guess it's due to the Hudi MOR table cann't read the Log file during the Compaction phase,and I also noticed that Trino mentioned this issue, Below is the relational url : https://github.com/trinodb/trino/pull/14786 and https://github.com/trinodb/trino/pull/14786/commits/c16a82407be6870c3e25c83a202f654e8f8f4d01.

We hope to slove the following Problems:
1. Trino supports Hudi mor table snapshot query (i.e. can read rt tables);
2. when reading of the hudi mor table by Trino (version 417) or PrestoDB (version 0.280) will not result in the above data quality issues;

![image](https://github.com/apache/hudi/assets/5379274/4301e326-e557-450e-aa2d-b10f3da0fc2a)

![image](https://github.com/apache/hudi/assets/5379274/23773f1e-b5d5-4bb0-ae39-57931f1856c6)

**To Reproduce**
Steps to reproduce the behavior:

1.
2.
3.
4.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.13.0+

* Spark version :3.2.1

* Hive version :3.1.0

* Hadoop version :3.1.1

* Storage (HDFS/S3/GCS..) :HDFS

* Running on Docker? (yes/no) : no
* PrestoDB version: 0.280
* Trino version: 417

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.