REPARTITION In Bloom Index Causing Slow Down
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I am using bloom index where will finding the file it is repartition and sorting seen in the spark dag view causing the problem of executor failure even I have increase memory per executor to 8GB for only 10GB of data.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a partition path based on year month day and hour
2.Enable Bloom Index and using HudiDeltstreamer to incrementally load data from the above path
3. The size of data is 10GB in parquet partitioned between three hours.
4. Try writing the
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.3
* Spark version :3.3
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) :no
Num Executor= 15
Num Core Per Executor =6
Drive Memory = 6GB
Drive Core= 6
Memory per executor=8gb
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Hudi 0.12.3 Bloom Index write with Spark 3.3, the stated S3 layout, and the provided executor settings, then inspect the Spark DAG for the repartition and sort stages. Capture the missing write steps and stacktrace; the issue is done when the executor failure and slowdown have a confirmed cause and a validated fix or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100