apache / apache/uniffle

[FEATURE][SPARK] Support cancel async thread of handle blockEvent and rpc when writer is killed

Open
#1,264 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

When task is killed for stage cancel, another task attempt succeed or some other reasons, The `AddBlockEvent` handling and `sendShuffleData` still work.
Although `needCancelRequest` may cancel some work, but the `AddBlockEvent` in the blocking queue of threadPool still holds the shuffleblockdata, and so as to the rpc request that are already called but waiting for repsonse.

That will cause 3 problems:
1. We freeAll memory onece the task is killed, but the shuffleBlockData hold by the async thread still occupy memory
2. Many useless runnable related to the kille task are still working or wait to be executed
4. Currently`checkBlockSendResult` can not be interrupted, when the killed task caused by speculation is the last one of the shuffle map stage, it will block the next reduce stage scheduling

### Motivation

_No response_

### Describe the solution

1. Cancel all the runnable that are wait to be executed or blocked in waiting for rpc callback
2. Interrupt `checkBlockSendResult` immediately

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Search the Java codebase for AddBlockEvent, sendShuffleData, needCancelRequest, and checkBlockSendResult, then trace how killed Spark tasks reach the async queue and RPC callbacks. Start by reading those entry points and their existing cancellation tests, if present. Done means queued or blocked work is cancelled, held shuffle data can be released, and checkBlockSendResult is interrupted for killed tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.