apache / apache/hudi

[SUPPORT] Is this the expected number of S3 calls?

Open
#9,612 9 comments 0 reactions 0 assignees View on GitHub
area:aws issue:high-fs-calls priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi, we are using Hudi on AWS. We have noticed the following unexpected behavior.

A `SELECT * FROM table` creates a significant number of S3 calls:

```
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
|path |httpMethod|cnt|
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
|my_table/.hoodie |HEAD |5 |
|my_table/.hoodie/ |HEAD |5 |
|my_table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile |HEAD |5 |
|my_table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile/ |HEAD |5 |
|my_table/.hoodie/20221124035739002.replacecommit |GET |5 |
|my_table/.hoodie/20221127222955674.replacecommit |GET |5 |
|my_table/.hoodie/20221128000946056.replacecommit |GET |5 |
|my_table/.hoodie/20230203015652867.replacecommit |GET |5 |
|my_table/.hoodie/20230203034909027.replacecommit |GET |5 |
|my_table/.hoodie/20230323023115954.replacecommit |GET |5 |
|my_table/.hoodie/20230323024631265.replacecommit |GET |5 |
|my_table/.hoodie/20230323041457900.replacecommit |GET |5 |
|my_table/.hoodie/20230627223911673.replacecommit |GET |5 |
|my_table/.hoodie/20230706040420663.replacecommit |GET |5 |
|my_table/.hoodie/20230821012127985.replacecommit |GET |5 |
|my_table/.hoodie/20230821013120957.replacecommit |GET |5 |
|my_table/.hoodie/20230823042339397.replacecommit |GET |5 |
|my_table/.hoodie/hoodie.properties |GET |5 |
|my_table/site_id%253D21/42d99963-db7f-400f-9e33-d539c74672aa-0_0-79-6549_20230323023115954.parquet |GET |3 |
|my_table/site_id%253D22/431ca0d1-8af3-4a72-bd17-31f2cd7e97e9-0_0-39-5633_20230323023017903.parquet |GET |3 |
|my_table/site_id%253D23/36675fed-d8ab-4532-aedc-dddf0a32accb-0_1-80-6551_20230323024631265.parquet |GET |3 |
|my_table/site_id%253D24/20efa6e4-489c-4b1a-a474-5dc1731485ed-0_0-80-6551_20230323041457900.parquet |GET |3 |
|my_table/site_id%253D30/15bfe605-bced-4d17-b571-62ebe64c5e97-0_0-80-6552_20230823042339397.parquet |GET |3 |
|my_table/site_id%253D30/27d907a2-7485-450d-9cdd-9f9c7e95fe88-0_0-39-5633_20230823044858848.parquet |GET |3 |
|my_table/site_id%253D21/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D21/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D21/42d99963-db7f-400f-9e33-d539c74672aa-0_0-79-6549_20230323023115954.parquet |HEAD |1 |
|my_table/site_id%253D22/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D22/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D22/431ca0d1-8af3-4a72-bd17-31f2cd7e97e9-0_0-39-5633_20230323023017903.parquet |HEAD |1 |
|my_table/site_id%253D23/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D23/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D23/36675fed-d8ab-4532-aedc-dddf0a32accb-0_1-80-6551_20230323024631265.parquet |HEAD |1 |
|my_table/site_id%253D24/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D24/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D24/20efa6e4-489c-4b1a-a474-5dc1731485ed-0_0-80-6551_20230323041457900.parquet |HEAD |1 |
|my_table/site_id%253D30/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D30/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D30/15bfe605-bced-4d17-b571-62ebe64c5e97-0_0-80-6552_20230823042339397.parquet |HEAD |1 |
|my_table/site_id%253D30/27d907a2-7485-450d-9cdd-9f9c7e95fe88-0_0-39-5633_20230823044858848.parquet |HEAD |1 |
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
```

Why are there so many `HEAD` calls?
Why are there multiple `GET` calls per object?

I'm creating this ticket because we have significant number of S3 calls across our Hudi tables which seem quite out of place given how many queries we do. They are starting to have non-negligible cost implications and even managed to cause throttling on S3 which impacted the Hudi job runs.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by tracing how a SELECT * query produces S3 HEAD and GET requests in Hudi, then compare the observed calls with the expected metadata and data reads. Done means explaining the repeated requests and identifying any confirmed cause or actionable reduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.