MPP query fails when a consumer finishes early: in-flight writes hit an already-closed tunnel and abort the query
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
1. Minimal reproduce step (Required)
Any MPP query in which an exchange consumer task legitimately finishes before
its upstream producer tasks have finished sending, while the producers still
have packets in flight toward it.
Known triggers:
- A hash join task whose build side is empty skips reading its probe input
and immediately closes its probe ExchangeReceiver, while the probe-side
producer tasks are still writing (the skip optimization from #11001 makes
this window wide and reliable, which is how the bug surfaced). - The same class of problem was previously observed with LIMIT-style early
exits (#7177). #7969 removed the gather-level error cascade, but the
producer-side write failure and the task-level error-packet cascade remain.
The failure itself is a timing race: it requires a producer write to land
after the consumer has already closed. On an unmodified build it occurs with
low probability; it can be reproduced deterministically with a small
delay-injection patch that stretches the producer write window.
2. What did you expect to see? (Required)
The query succeeds. A consumer legitimately finishing early — because of
LIMIT, an empty build side, a filter that eliminates all rows, etc. — is a
fact that upstream producers must tolerate, not an error. In-flight packets
that arrive after the consumer has finished should be discarded silently,
and the producer tasks should finish normally.
3. What did you see instead (Required)
The query fails with err msg like:
ERROR 1105 (HY000): write to tunnel tunnel13+21 which is already closed, tunnel13+21: unexpectedWriteDone called
The correct query result is lost. Root cause:
- A consumer task legitimately finishes early and closes its
ExchangeReceiver / MPPTunnel while upstream producers are still sending. - The MPP exchange layer has no graceful-close handshake: in the async gRPC
path the close reason (clean vs. erroneous) is not transported, so the
producer cannot distinguish a clean consumer finish from a real failure,
and its in-flight write fails with "tunnel is already closed" /
"unexpectedWriteDone called". - The failed producer task reports error packets, which cascade to sibling
tasks and abort the whole MPP query, even though every task's own work
was correct.
Note this weakness is not introduced by #11001; it has existed in the
exchange layer since the beginning. #11001 merely added a new early-finisher
(the empty-build skip) that hits the window reliably.
4. What is your TiFlash version? (Required)
master (exchange-layer weakness predates #11001; reliably triggered by it)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the MPP exchange layer, especially ExchangeReceiver, MPPTunnel, and the async gRPC path described in the report. Reproduce the early-consumer finish with delay injection, then trace how a closed tunnel becomes a producer and task error; done means LIMIT, empty-build, and similar early exits complete successfully without error-packet cascades.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100