[SUPPORT] Is this the expected number of S3 calls?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Hi, we are using Hudi on AWS. We have noticed the following unexpected behavior.
A `SELECT * FROM table` creates a significant number of S3 calls:
```
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
|path |httpMethod|cnt|
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
|my_table/.hoodie |HEAD |5 |
|my_table/.hoodie/ |HEAD |5 |
|my_table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile |HEAD |5 |
|my_table/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile/ |HEAD |5 |
|my_table/.hoodie/20221124035739002.replacecommit |GET |5 |
|my_table/.hoodie/20221127222955674.replacecommit |GET |5 |
|my_table/.hoodie/20221128000946056.replacecommit |GET |5 |
|my_table/.hoodie/20230203015652867.replacecommit |GET |5 |
|my_table/.hoodie/20230203034909027.replacecommit |GET |5 |
|my_table/.hoodie/20230323023115954.replacecommit |GET |5 |
|my_table/.hoodie/20230323024631265.replacecommit |GET |5 |
|my_table/.hoodie/20230323041457900.replacecommit |GET |5 |
|my_table/.hoodie/20230627223911673.replacecommit |GET |5 |
|my_table/.hoodie/20230706040420663.replacecommit |GET |5 |
|my_table/.hoodie/20230821012127985.replacecommit |GET |5 |
|my_table/.hoodie/20230821013120957.replacecommit |GET |5 |
|my_table/.hoodie/20230823042339397.replacecommit |GET |5 |
|my_table/.hoodie/hoodie.properties |GET |5 |
|my_table/site_id%253D21/42d99963-db7f-400f-9e33-d539c74672aa-0_0-79-6549_20230323023115954.parquet |GET |3 |
|my_table/site_id%253D22/431ca0d1-8af3-4a72-bd17-31f2cd7e97e9-0_0-39-5633_20230323023017903.parquet |GET |3 |
|my_table/site_id%253D23/36675fed-d8ab-4532-aedc-dddf0a32accb-0_1-80-6551_20230323024631265.parquet |GET |3 |
|my_table/site_id%253D24/20efa6e4-489c-4b1a-a474-5dc1731485ed-0_0-80-6551_20230323041457900.parquet |GET |3 |
|my_table/site_id%253D30/15bfe605-bced-4d17-b571-62ebe64c5e97-0_0-80-6552_20230823042339397.parquet |GET |3 |
|my_table/site_id%253D30/27d907a2-7485-450d-9cdd-9f9c7e95fe88-0_0-39-5633_20230823044858848.parquet |GET |3 |
|my_table/site_id%253D21/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D21/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D21/42d99963-db7f-400f-9e33-d539c74672aa-0_0-79-6549_20230323023115954.parquet |HEAD |1 |
|my_table/site_id%253D22/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D22/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D22/431ca0d1-8af3-4a72-bd17-31f2cd7e97e9-0_0-39-5633_20230323023017903.parquet |HEAD |1 |
|my_table/site_id%253D23/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D23/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D23/36675fed-d8ab-4532-aedc-dddf0a32accb-0_1-80-6551_20230323024631265.parquet |HEAD |1 |
|my_table/site_id%253D24/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D24/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D24/20efa6e4-489c-4b1a-a474-5dc1731485ed-0_0-80-6551_20230323041457900.parquet |HEAD |1 |
|my_table/site_id%253D30/.hoodie_partition_metadata |HEAD |1 |
|my_table/site_id%253D30/.hoodie_partition_metadata |GET |1 |
|my_table/site_id%253D30/15bfe605-bced-4d17-b571-62ebe64c5e97-0_0-80-6552_20230823042339397.parquet |HEAD |1 |
|my_table/site_id%253D30/27d907a2-7485-450d-9cdd-9f9c7e95fe88-0_0-39-5633_20230823044858848.parquet |HEAD |1 |
+---------------------------------------------------------------------------------------------------------------------------------+----------+---+
```
Why are there so many `HEAD` calls?
Why are there multiple `GET` calls per object?
I'm creating this ticket because we have significant number of S3 calls across our Hudi tables which seem quite out of place given how many queries we do. They are starting to have non-negligible cost implications and even managed to cause throttling on S3 which impacted the Hudi job runs.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by tracing how a SELECT * query produces S3 HEAD and GET requests in Hudi, then compare the observed calls with the expected metadata and data reads. Done means explaining the repeated requests and identifying any confirmed cause or actionable reduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100