apache / apache/uniffle

[DOCS] shuffle-server with low-memory best practice

Open
#1,169 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

I think we need to describe more sections to how to configure the client/server config on a low memory config.

From my point of view, once you configure lower memory, you have to accept the slowdown caused by back pressure when running more. As we all know, Uniffle uses memory as a high-speed buffer, but once the buffer is full, the rate of receiving shuffle data will be reduced to the highest write rate of the disk, so it will be back pressured.

In this case, we have two ways to avoid task failure
1. make the lower watermark for memory store. such as ` rss.server.high.watermark.write 20` , `rss.server.low.watermark.write 10`
2. increase the client timeout. `rss.client.send.check.timeout.ms 1200000`

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing shuffle-server and client configuration documentation and read the sections covering memory watermarks, disk back pressure, and timeouts. Document the low-memory tradeoff, including rss.server.high.watermark.write, rss.server.low.watermark.write, and rss.client.send.check.timeout.ms, and explain how these settings help avoid task failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.