apache / apache/hudi

Partitioning data into two keys is taking more time (10x) than partitioning into one key.

Open
#10,456 28 comments 1 reaction 0 assignees View on GitHub
area:performance engine:flink priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am trying to add second level of partition to my table instead of one level of partition but it is taking 10X time as compared to single level partition in hudi flink job.

I tried to ingest 1.8M record into one level of partition and it took around 12-15 Min to ingest all the data then with same configuration I just added another level of partition key with same data payload and it took around 1 hour 45 Min to complete the process.

**To Reproduce**

Steps to reproduce the behavior:
below is the configuration that I am using for table. You can add the table creation statement with below properties.

```
PARTITIONED BY (`client_id`,`hashed_server_id`)
WITH ('connector' = 'hudi','path' = '${table_location}',
'table.type' = 'COPY_ON_WRITE',
'hoodie.datasource.write.recordkey.field' = 'a,b',
'payload.class'='x.y.PartialUpdate',
'precombine.field'='ts',
'hoodie.clean.async'='true',
'hoodie.cleaner.policy' = 'KEEP_LATEST_COMMITS',
'hoodie.clean.automatic' = 'true',
'hoodie.clean.max.commits'='5',
'hoodie.clean.trigger.strategy'='NUM_COMMITS',
'hoodie.cleaner.parallelism'='100',
'hoodie.cleaner.commits.retained'='4',
'hoodie.index.type'= 'BUCKET',
'hoodie.index.bucket.engine' = 'SIMPLE',
'hoodie.bucket.index.num.buckets'='16',
'hoodie.bucket.index.hash.field'='a',
'hoodie.parquet.small.file.limit'='104857600',
'hoodie.parquet.compression.codec'='snappy')
```

**Expected behavior**
As it is just a partition addition to the storage it should not impact the performance much(I can understand if it takes 5-7 min extra as complexkey generation is bit slower than simplekey ).

**Environment Description**
* Flink 1.17.1
* Hudi version : 14

* Spark version : NA

* Hive version : NA

* Hadoop version : 3.4.0

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) :Yes

**Additional context**

My table type is upsert and I have test the functionality and it is working fine and I cannot change the table type.

I also discussed with @ad1happy2go and he also suggested that it wont impact much as it just a another level of partition.

CC : @ad1happy2go @codope @danny0405 @yo

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the comparison with the provided Flink table definition, using one partition key and then both client_id and hashed_server_id. Compare ingestion timing and relevant job logs for the 1.8M-record workload under the stated Hudi, S3, and Docker environment. Done means identifying the cause of the regression and documenting or fixing it so the two-level partition case no longer incurs the reported slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, java
Domain
cloud, data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.