apache / apache/uniffle

[Bug] The block that the Task fails to send needs to be cleared

Open
#1,105 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

When a Task sends a Block to the ShuffleServer, it is placed in the DataPusher asynchronous thread to send it, but if the Task fails before the block is finished sending, the block to be sent is not cleaned.

### Affects Version(s)

0.8.0

### Uniffle Server Log Output

```logtalk
2023-08-07 10:41:16,838 INFO [Executor task launch worker for task 0.0 in stage 2.0 (TID 10)] writer.RssShuffleWriter (RssShuffleWriter.java:checkBlockSendResult(273)) - Wait 327 blocks sent to shuffle server
2023-08-07 10:41:16,838 INFO [Executor task launch worker for task 1.0 in stage 2.0 (TID 11)] writer.RssShuffleWriter (RssShuffleWriter.java:checkBlockSendResult(273)) - Wait 350 blocks sent to shuffle server
```

### Uniffle Engine Log Output

_No response_

### Uniffle Server Configurations

_No response_

### Uniffle Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at RssShuffleWriter.java:checkBlockSendResult(273) and trace how the DataPusher asynchronous thread tracks blocks while a Task is failing. Reproduce the failure path and verify that an unfinished block is removed instead of remaining queued for sending.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.