[SUPPORT] AWS Glue 4.0 taking to long to write to S3
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I ran a test reading 1 day worth of data vs 30 days worth of data from the external table to s3 in hudi format, and the amount of time it takes are roughly the same. I have also changed the write operation from upsert to insert, and it made no difference.
The file is not that large, each day has about 1m records from the external table. In s3, each file has an average size of 25MB partition by the date and hour. The whole process is copying and writing to s3, so there aren't any transformation process in between beside changing the data type.
Is there anything I can do to improve the speed?
```
hudi_options = {
'hoodie.table.name': table_name,
"hoodie.datasource.write.storage.type": "COPY_ON_WRITE",
'hoodie.datasource.write.recordkey.field': 'recordkey1, recordkey2',
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.operation': 'insert',
'hoodie.datasource.write.precombine.field': 'timestamp',
'hoodie.datasource.write.partitionpath.field': 'date, hour',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.datasource.hive_sync.partition_fields': 'date, hour',
'hoodie.datasource.hive_sync.enable': 'true',
"hoodie.datasource.hive_sync.mode":"hms",
'hoodie.datasource.hive_sync.sync_as_datasource': 'false',
'hoodie.datasource.hive_sync.database': database_name,
'hoodie.datasource.hive_sync.table': table_name,
'hoodie.datasource.hive_sync.use_jdbc': 'false',
'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.hive_sync.support_timestamp': 'true',
'hoodie.cleaner.policy': 'KEEP_LATEST_FILE_VERSIONS',
'hoodie.clustering.plan.strategy.max.bytes.per.group': '107374182400',
'hoodie.clustering.plan.strategy.max.num.groups': '1'
}
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.3
* Hive version : not sure
* Hadoop version : not sure
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the AWS Glue/Spark job and review the hudi_options configuration, especially the write, partitioning, clustering, and Hive sync settings. Reproduce the comparison between one day and 30 days of data, then identify and document a configuration or processing change that measurably improves S3 write time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cloud, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100