apache / apache/hudi

too many s3 list when hoodie.metadata.enable=true

Open
#9,751 27 comments 0 reactions 0 assignees View on GitHub
area:index area:metadata-table issue:high-fs-calls priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I upgrade hudi from **0.7 to 0.13.1 (**actually just replace spark hudi bundle jar). In hudi 0.13.1 metadata enabled is by default. But why there are still also so many list operations and it is not reduced . I can see the metadata/files are generated and also mentioned that Listed files in partition from metadata. Did I miss any config? Why there are still so many s3 list operations

image


**Environment Description**

* Hudi version : 0.13.1

* Spark version : 3.3.2

* Hive version : 3

* Hadoop version : 3.2.2

* Storage (HDFS/S3/GCS..) : s3

* Running on Docker? (yes/no) : no

I am using the third party's s3 storage(local deployed storage cluster) and they provided me the s3 list operations metrics.
I am using default config.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported S3 list-operation behavior with Hudi 0.13.1, Spark 3.3.2, metadata enabled, and the listed Hadoop and S3-compatible environment. Compare the S3 metrics with Hudi's metadata/files output and configuration, then establish whether the remaining list operations are expected or indicate a defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.