apache / apache/uniffle

[FEATURE] Print out the task shuffle write time statistics by table format in log

Open
#1,426 11 comments 0 reactions 1 assignee Claimed by @myandpr View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Currently, we can't find out the slow shuffle-server in the client side log, this is not ease to use, especially for server with terriable GC.

I hope the metric with the server dimension for one spark task could be shown like the following table format in the client side log.

| | Min | 25th percentile | Median | 75th percentile | Max |
|------------------------------|:-----------------:|:------------------:|:-----------------:|--------------------|--------------------|
| Shuffle Write Duration | 10s | 15s | 25s | 30s | 4min |
| Shuffle Write Size / Records | 20M / 10000 | 20M / 10000 | 20M / 10000 | 20M / 10000 | 20M / 10000 |
| Shuffle server list | 10.23.35.19-21001 | 10.23.35.134-21001 | 10.23.35.14-21001 | 10.23.315.14-21001 | 10.123.35.14-21001 |

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.