apache / apache/hudi

[SUPPORT] Flink COW upsert is too slow to sink the data to Hudi-S3

Open
#12,951 16 comments 0 reactions 0 assignees View on GitHub
area:performance engine:flink
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Flink + Hudi COW + upsert mode + S3 path -> too slow. Nearly taking half an hour for 13k records only.

Even MOR is very slow. What I have noticed is that flink is slow, while running the same process on AWS glue is comparatively fast. With same configs, it should give take same time to process

**To Reproduce**

Steps to reproduce the behavior:

Use these configuration to reproduce
" 'connector' = 'hudi'," +
" 'path' = '" + S3_PATH + "'," +
" 'table.type' = 'COPY_ON_WRITE'," +
" 'hoodie.table.name' = '" + HUDI_TABLE_NAME + "'," +
" 'hoodie.datasource.write.recordkey.field' = 'event_id'," +
" 'hoodie.datasource.write.partitionpath.field' = 'customer_uuid,year,month'," +
" 'hoodie.datasource.write.precombine.field' = 'time'," +
" 'hoodie.schema.on.read.enable' = 'true'," +
" 'hoodie.parquet.compression.codec' = 'snappy'," +
" 'hoodie.datasource.write.operation' = 'upsert',"+
" 'hoodie.parquet.max.file.size' = '10485760',"+
" 'write.tasks' = '8',"+
" 'hoodie.populate.meta.fields' = 'true',"+
" 'hoodie.table.keygenerator.class' = 'org.apache.hudi.keygen.NonpartitionedKeyGenerator',"+
" 'hoodie.metadata.enable' = 'false'"+

**Expected behavior**

It should process the data within a minutes, evne for 1 million records.

**Environment Description**

* Hudi version : 0.15.0

* Flink version : 1.18

* Hadoop version : 3.1.4

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional context**

- use these configs

```
" 'connector' = 'hudi'," +
" 'path' = '" + S3_PATH + "'," +
" 'table.type' = 'COPY_ON_WRITE'," +
" 'hoodie.table.name' = '" + HUDI_TABLE_NAME + "'," +
" 'hoodie.datasource.write.recordkey.field' = 'event_id'," +
" 'hoodie.datasource.write.partitionpath.field' = 'customer_uuid,year,month'," +
" 'hoodie.datasource.write.precombine.field' = 'time'," +
" 'hoodie.schema.on.read.enable' = 'true'," +
" 'hoodie.parquet.compression.codec' = 'snappy'," +
" 'hoodie.datasource.write.operation' = 'upsert',"+
" 'hoodie.parquet.max.file.size' = '10485760',"+
" 'write.tasks' = '8',"+
" 'hoodie.populate.meta.fields' = 'true',"+
" 'hoodie.table.keygenerator.class' = 'org.apache.hudi.keygen.NonpartitionedKeyGenerator',"+
" 'hoodie.metadata.enable' = 'false'"+
```

- Sinking time is very random but it is taking ~half an hour for 13k and for 28K it tool 2 hours, which is very wierd

![Image](https://github.com/user-attachments/assets/1618785e-3395-4bf6-9ed5-eaaca1dd731d)

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are identified. Start by reproducing the Flink 1.18 write with the listed Hudi 0.15.0 settings against S3, compare it with AWS Glue, and use the timing variation to identify the bottleneck; done means 13K and 28K records complete within the expected minutes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.