[DOCS] shuffle-server with low-memory best practice
- Dominant language
- Java
- Stars
- 454
- Forks
- 172
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 5
Description
I think we need to describe more sections to how to configure the client/server config on a low memory config.
From my point of view, once you configure lower memory, you have to accept the slowdown caused by back pressure when running more. As we all know, Uniffle uses memory as a high-speed buffer, but once the buffer is full, the rate of receiving shuffle data will be reduced to the highest write rate of the disk, so it will be back pressured.
In this case, we have two ways to avoid task failure
1. make the lower watermark for memory store. such as ` rss.server.high.watermark.write 20` , `rss.server.low.watermark.write 10`
2. increase the client timeout. `rss.client.send.check.timeout.ms 1200000`
Contributor guide
Research direction
Start by locating the existing shuffle-server and client configuration documentation and read the sections covering memory watermarks, disk back pressure, and timeouts. Document the low-memory tradeoff, including rss.server.high.watermark.write, rss.server.low.watermark.write, and rss.client.send.check.timeout.ms, and explain how these settings help avoid task failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100