OOM issue
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We get lots of OOM errors in spark test, like: "
4545568 [Executor task launch worker for task 18] ERROR org.apache.spark.executor.Executor [] - Exception in task 2.0 in stage 11.0 (TID 18)
[10170|https://github.com/apache/hudi/actions/runs/7458499094/job/20292582957?pr=10464#step:6:10171]org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 188 bytes of memory, got 0",
Which blocks the all Spark tests already.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7283
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-4302
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Spark test failures and reviewing the linked GitHub Actions job for the OOM context. Trace the failing Spark tests and use the HUIDI-7283 JIRA sub-task for additional scope; done means the Spark test suite completes without these out-of-memory errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100