matrixorigin / matrixorigin/matrixone
[Bug] dev:distributed join aborts when remote dispatch targets another local pipeline tree
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
A valid distributed recursive-CTE join was aborted by the MatrixOne remote execution pipeline. The client received an internal `20101 context canceled` error although the configured statement timeout was 3600 seconds and the statement had run for about 0.38 seconds.
The earliest server-side evidence for the same trace reports that a pipeline could not execute remotely because its dispatch operator targeted another local pipeline tree. This is followed by remote receiver closure and an aborted shuffle pool.
## Environment
- Deployment: cn-dev `freetier-01`, multi-CN distributed deployment; profile `s16c64g`
- Regression workflow: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/34021248084
- Failed job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/34021248084/job/101476204876
- Workflow source revision: `76de212c3bc05892e32c7cf436706297c78ee6eb`
- Observation time: 2026-09-06 13:26:16 UTC
- Concurrent workload: NESR continuous writer was enabled
This report is a 1/1 reproduction on the current cn-dev deployment. It does not claim reproduction on official `main`; that verification is still required.
## Steps to reproduce
1. Load the Jinpan `jinpan_001` dataset into a multi-CN MatrixOne deployment.
2. Execute the `@q0261_payment_details_tree` statement from [`j_batch_014.sql`](https://github.com/matrixorigin/mo-nightly-regression/blob/76de212c3bc05892e32c7cf436706297c78ee6eb/tools/mo-regression-test/cases/jinpan_001/join/j_batch_014.sql), with the concurrent NESR writer enabled.
3. In the observed run, the statement started at `13:26:16.242 UTC` and failed at `13:26:16.625 UTC`.
## Actual behavior
The client received:
```text
(20101, 'internal error: convert go error to mo error context canceled')
```
The same trace ID, `61cf8d08-9613-d1ec-213e-2f27969d0b3b`, was present on participating CNs `cn-rm8h5` and `cn-5w5sf`.
The first relevant server-side messages include:
```text
the pipeline ... cannot execute remotely because its dispatch operator targets another local pipeline tree
shuffle pool was aborted before prepare completed
invalid state remote dispatch receiver ... is already closed
query interrupted
```
The failure then propagated to the client as `context canceled`.
## Expected behavior
The valid statement should complete successfully, or return a specific, actionable execution error. A remote dispatch/shuffle topology inconsistency must not surface as the generic internal `20101 context canceled` error.
## Excluded causes
- Not a statement timeout: the run configured a 3600-second timeout and this statement failed in approximately 0.38 seconds.
- Not an OOM or process crash: no `OOMKilled`, exit 137, panic, or fatal error was found in the event window.
- Not CN lifecycle disruption: the two participating CNs had zero restart increments in the surrounding 10-minute window; no Draining, Pod deletion, or session migration event was recorded. The coordinator session logged `migrate-goRoutineId=0`.
- Not a runner disconnect: the runner reconnected and continued with later SQL files after this file was marked failed.
## Evidence
- Workflow/job links above.
- Trace ID: `61cf8d08-9613-d1ec-213e-2f27969d0b3b`.
- Server timestamps: `2026-09-06 13:26:16.299–13:26:16.420 UTC`.
## Related
This is related in symptom to #27776 (remote work interruption), but the observed window contains no CN Draining/reclaim event, so it should be investigated independently.
Contributor guide
Assessment
This issue has not been assessed yet.