apache / apache/hudi

Trino Hive Connector: Hudi Metadata Performance Regression

Open
#16,235 1 comment 0 reactions 1 assignee Assigned to @rahil-c View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When using Athena v3 (Trino) to query a hudi table with 10k partitions noticed very slow query performance, compared to when disabling metadata feature. 

It seems the bottleneck happens to be around when multiple trino hive thread attept reading of HFile data, are stuck in waiting states due to having to obtain lock. 

Noticed that disabling following config

```

Disabling {{CACHE_DATA_ON_READ}} for HFileReader in {{HoodieHFileReaderFactory}}

```

can increase query perf when using metadata enabled but will need to do more investigation as to whether this has any side effect. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6876
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.