apache / apache/hudi

[SUPPORT] How to configure spark and flink to write mor tables using bucket indexes?

Open
#11,946 12 comments 0 reactions 0 assignees View on GitHub
area:index area:performance
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I want to use flink and spark to write to the mor table, and use bucket CONSISTENT_HASHING for the index, but I find that spark is very fast to write the full amount and flink is very slow(flink write 100record/s) to write increments.
spark sql:
```
CREATE TABLE test.tableA ()
USING hudi
TBLPROPERTIES (
'connector' = 'hudi',
'index.type'='BUCKET',
'hoodie.index.type'='BUCKET',
'hoodie.index.bucket.engine'='CONSISTENT_HASHING',
'hoodie.datasource.write.recordkey.field' = '',
'path' = '',
'preCombineField' = 'create_time',
'precombine.field' = 'create_time',
'primaryKey' = '',
'table.type' = 'MERGE_ON_READ',
'write.rate.limit'='10000',--flink配置
'write.tasks'='2',--flink配置
'write.utc-timezone'='false',
'type' = 'mor');
```
flink_slow How to optimize?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Spark SQL definition and Flink settings, focusing on the MOR table, CONSISTENT_HASHING bucket index, write rate limit, and task count. Reproduce a controlled incremental-write comparison and use the reported throughput to identify the limiting configuration or behavior; done means an evidence-based optimization or explanation for the Flink performance gap.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, performance, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.