apache / apache/hudi

REPARTITION In Bloom Index Causing Slow Down

Open
#11,888 5 comments 0 reactions 0 assignees View on GitHub
area:performance
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I am using bloom index where will finding the file it is repartition and sorting seen in the spark dag view causing the problem of executor failure even I have increase memory per executor to 8GB for only 10GB of data.

**To Reproduce**

Steps to reproduce the behavior:

1. Create a partition path based on year month day and hour
2.Enable Bloom Index and using HudiDeltstreamer to incrementally load data from the above path
3. The size of data is 10GB in parquet partitioned between three hours.
4. Try writing the

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.12.3

* Spark version :3.3

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) : s3

* Running on Docker? (yes/no) :no

Num Executor= 15
Num Core Per Executor =6
Drive Memory = 6GB
Drive Core= 6
Memory per executor=8gb
**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Hudi 0.12.3 Bloom Index write with Spark 3.3, the stated S3 layout, and the provided executor settings, then inspect the Spark DAG for the repartition and sort stages. Capture the missing write steps and stacktrace; the issue is done when the executor failure and slowdown have a confirmed cause and a validated fix or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.