apache / apache/uniffle

[Improvement] When committing to a shuffle server, print its IP and port

Open
#1,614 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

Currently, we don't know which shuffle server the client is committing to.
```
[03:08:14:067] [Executor task launch worker for task 18899.0 in stage 3.0 (TID 64760)] INFO org.apache.spark.shuffle.writer.RssShuffleWriter.sendCommit:555 - Wait commit to shuffle server for task[75596] cost 16201 ms
[03:08:19:067] [Executor task launch worker for task 18899.0 in stage 3.0 (TID 64760)] INFO org.apache.spark.shuffle.writer.RssShuffleWriter.sendCommit:555 - Wait commit to shuffle server for task[75596] cost 21202 ms
[03:08:24:067] [Executor task launch worker for task 18899.0 in stage 3.0 (TID 64760)] INFO org.apache.spark.shuffle.writer.RssShuffleWriter.sendCommit:555 - Wait commit to shuffle server for task[75596] cost 26202 ms
[03:08:29:067] [Executor task launch worker for task 18899.0 in stage 3.0 (TID 64760)] INFO org.apache.spark.shuffle.writer.RssShuffleWriter.sendCommit:555 - Wait commit to shuffle server for task[75596] cost 31202 ms
```
We can add the shuffle server's IP and port into the log.

### How should we improve?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at RssShuffleWriter.sendCommit, where the existing wait-commit log is emitted. Trace the shuffle server target available at that point and include its IP and port in the message; the work is done when the commit log identifies the server. No test file is named in the issue, so inspect the existing tests around this entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.