apache / apache/uniffle

[FEATURE] Better memory limitation when overlapping decompression is enabled

Open
#2,716 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Currently, the overlapping decompression mechanism may cause Spark jobs to run out of memory due to the lack of memory limits.

In the first phase, we can introduce limits on the number of staging buffers. Subsequently, we can align with Spark’s memory consumer model to precisely request and release memory.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Locate the overlapping decompression mechanism and inspect how staging buffers are created and retained. Compare the current behavior with Spark’s memory consumer model, while initially focusing on a limit for the number of staging buffers. Done means overlapping decompression has a bounded buffer count and tests demonstrate that Spark jobs no longer exceed memory because of unbounded staging buffers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.