Trino Hive Connector: Hudi Metadata Performance Regression
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When using Athena v3 (Trino) to query a hudi table with 10k partitions noticed very slow query performance, compared to when disabling metadata feature.
It seems the bottleneck happens to be around when multiple trino hive thread attept reading of HFile data, are stuck in waiting states due to having to obtain lock.
Noticed that disabling following config
```
Disabling {{CACHE_DATA_ON_READ}} for HFileReader in {{HoodieHFileReaderFactory}}
```
can increase query perf when using metadata enabled but will need to do more investigation as to whether this has any side effect.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6876
- Type: Bug
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.