[FEATURE] Better memory limitation when overlapping decompression is enabled
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Currently, the overlapping decompression mechanism may cause Spark jobs to run out of memory due to the lack of memory limits.
In the first phase, we can introduce limits on the number of staging buffers. Subsequently, we can align with Spark’s memory consumer model to precisely request and release memory.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
Contributor guide
Research direction
Locate the overlapping decompression mechanism and inspect how staging buffers are created and retained. Compare the current behavior with Spark’s memory consumer model, while initially focusing on a limit for the number of staging buffers. Done means overlapping decompression has a bounded buffer count and tests demonstrate that Spark jobs no longer exceed memory because of unbounded staging buffers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100