MPP dispatch/cancel race can leave accepted TiFlash tasks without cancellation
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### Summary
An MPP dispatch error can race with `dispatchAll()` so that a task is dispatched after the coordinator has already taken its one-time cancellation snapshot. The task's store may be absent from that snapshot, while later cancellation attempts are suppressed because every request has already been marked `MppTaskCancelled`.
Separately, `MPPClient.CancelMPPTasks()` sends one cancel RPC per selected store and does not retry a transport failure.
These are TiDB cancellation reliability gaps. After a full incident-log review, however, they are **not confirmed as the cause of the customer backlog** described in pingcap/tiflash#11059.
### Affected version
- Release: `CLOUD.202603.10`
- Commit: `6b4c9f9dce2118aca580a16ef49bd791f4e12b3a`
### Source-level race
`dispatchAll()` originally launches `handleDispatchReq()` even when cancellation changed the request from `MppTaskReady` to `MppTaskCancelled` before the loop reached it.
At the same time, `cancelMppTasks()`:
1. includes only stores represented by a `MppTaskRunning` request;
2. marks every request `MppTaskCancelled`; and
3. returns immediately on every later call.
The following interleaving is therefore reachable:
1. task A starts dispatch and returns an error;
2. cancellation snapshots only the stores currently marked running and marks task B cancelled;
3. `dispatchAll()` still launches task B;
4. no later cancellation is sent for task B's store.
Relevant code:
- [`dispatchAll()` and `sendError()`](https://github.com/pingcap/tidb/blob/6b4c9f9dce2118aca580a16ef49bd791f4e12b3a/pkg/executor/internal/mpp/local_mpp_coordinator.go#L582-L615)
- [`cancelMppTasks()`](https://github.com/pingcap/tidb/blob/6b4c9f9dce2118aca580a16ef49bd791f4e12b3a/pkg/executor/internal/mpp/local_mpp_coordinator.go#L704-L727)
- [`MPPClient.CancelMPPTasks()`](https://github.com/pingcap/tidb/blob/6b4c9f9dce2118aca580a16ef49bd791f4e12b3a/pkg/store/copr/mpp.go#L193-L231)
### Production reachability and evidence boundary
For `query_ts=1787741714677986579`, the incident logs show a cancel reaching TiFlash at 03:55:14.736 PDT and a late dispatch for task 6 being accepted at 03:55:14.749. TiFlash then aborted the task and recorded `CANCELLED` with an end timestamp at 03:55:14.764.
This proves that cancel/late-dispatch interleaving is reachable, but this particular task was cleaned successfully.
The strongest retained-task sample is now tracked in pingcap/tiflash#11059: cancel reached TiFlash and `Finish abort task from running` was logged, but the task remained alive for more than 11 hours. That is a TiFlash execution-cancellation problem, not evidence that this TiDB race caused the retained population.
### Expected behavior
- Do not launch a dispatch for a request already marked cancelled.
- Ensure every store which may accept a dispatch is covered by idempotent cancellation.
- Retry or otherwise reconcile transient cancel delivery failures per store.
- Track cancellation delivery separately from the local `MppTaskCancelled` state.
Related TiFlash cleanup issue: [pingcap/tiflash#11059](https://github.com/pingcap/tiflash/issues/11059).
Contributor guide
Research direction
Start with dispatchAll() and sendError() in pkg/executor/internal/mpp/local_mpp_coordinator.go, then read cancelMppTasks() and MPPClient.CancelMPPTasks() in pkg/store/copr/mpp.go. Trace the dispatch/cancellation interleaving and the per-store cancel delivery path. Done means canceled requests are not dispatched, every potentially accepting store is covered, and transient cancel delivery is reconciled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100