apache / apache/hudi

[SUPPORT] AWS Glue 4.0 taking to long to write to S3

Open
#10,916 3 comments 0 reactions 0 assignees View on GitHub
area:aws area:performance priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I ran a test reading 1 day worth of data vs 30 days worth of data from the external table to s3 in hudi format, and the amount of time it takes are roughly the same. I have also changed the write operation from upsert to insert, and it made no difference.

The file is not that large, each day has about 1m records from the external table. In s3, each file has an average size of 25MB partition by the date and hour. The whole process is copying and writing to s3, so there aren't any transformation process in between beside changing the data type.

Is there anything I can do to improve the speed?

```
hudi_options = {
'hoodie.table.name': table_name,
"hoodie.datasource.write.storage.type": "COPY_ON_WRITE",
'hoodie.datasource.write.recordkey.field': 'recordkey1, recordkey2',
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.operation': 'insert',
'hoodie.datasource.write.precombine.field': 'timestamp',
'hoodie.datasource.write.partitionpath.field': 'date, hour',
'hoodie.datasource.write.hive_style_partitioning': 'true',

'hoodie.datasource.hive_sync.partition_fields': 'date, hour',
'hoodie.datasource.hive_sync.enable': 'true',
"hoodie.datasource.hive_sync.mode":"hms",
'hoodie.datasource.hive_sync.sync_as_datasource': 'false',
'hoodie.datasource.hive_sync.database': database_name,
'hoodie.datasource.hive_sync.table': table_name,
'hoodie.datasource.hive_sync.use_jdbc': 'false',
'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.hive_sync.support_timestamp': 'true',

'hoodie.cleaner.policy': 'KEEP_LATEST_FILE_VERSIONS',
'hoodie.clustering.plan.strategy.max.bytes.per.group': '107374182400',
'hoodie.clustering.plan.strategy.max.num.groups': '1'

}
```
**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.12.1

* Spark version : 3.3

* Hive version : not sure

* Hadoop version : not sure

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

Screenshot 2024-03-23 at 14 33 41

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the AWS Glue/Spark job and review the hudi_options configuration, especially the write, partitioning, clustering, and Hive sync settings. Reproduce the comparison between one day and 30 days of data, then identify and document a configuration or processing change that measurably improves S3 write time.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, spark
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.