too many s3 list when hoodie.metadata.enable=true
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I upgrade hudi from **0.7 to 0.13.1 (**actually just replace spark hudi bundle jar). In hudi 0.13.1 metadata enabled is by default. But why there are still also so many list operations and it is not reduced . I can see the metadata/files are generated and also mentioned that Listed files in partition from metadata. Did I miss any config? Why there are still so many s3 list operations
**Environment Description**
* Hudi version : 0.13.1
* Spark version : 3.3.2
* Hive version : 3
* Hadoop version : 3.2.2
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
I am using the third party's s3 storage(local deployed storage cluster) and they provided me the s3 list operations metrics.
I am using default config.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported S3 list-operation behavior with Hudi 0.13.1, Spark 3.3.2, metadata enabled, and the listed Hadoop and S3-compatible environment. Compare the S3 metrics with Hudi's metadata/files output and configuration, then establish whether the remaining list operations are expected or indicate a defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100