matrixorigin / matrixorigin/matrixone
[Bug]: ISCP DDL drain lacks an ownership-generation quiescence proof
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
### Branch Name
`main` / follow-up to #26181 and #26183
### Commit ID
`ff0a3faf2b4c245b4682abf019a99ccb63e62568`
### Other Environment Information
- Deployment: Multi-CN with the ISCP daemon task able to fail over between CNs.
- Impact status: protocol-level correctness gap. It is not the assignment-before-executor-readiness failure reported in #26181, and no separate production incident is claimed here.
### Actual Behavior
`drainIndexCdcTaskConsumer` fences only the daemon-task owners that it observes. Its convergence condition accepts a fresh read of a previously fenced CN UUID.
This is not an ownership-generation linearization point. During a long DDL, ownership can transiently move from CN A to CN B and back to CN A between two owner reads, or move after the final read. CN B can start an ISCP consumer without ever receiving a drain request. The DDL can then complete with no proof that every consumer generation which ran during the operation was quiesced.
The current implementation improves the common readiness and observed-handoff paths: it retries startup readiness, re-reads owners, fences observed CNs, renews fences, and carries fences across executor replacement in one CN process. It intentionally does not provide a durable cross-process ownership proof.
### Expected Behavior
An index/CDC-affecting DDL must either:
1. obtain a quiescence proof tied to a stable daemon-task ownership generation; or
2. fail/retry when ownership changes before that proof is established.
A consumer must reject admission when its ownership generation is fenced, including after executor or CN restart. Stale renew/remove messages must not affect a newer generation.
### Steps to Reproduce
A deterministic integration or fault-injection test should model the following sequence:
1. Start a long ISCP-affecting DDL and let it observe and drain owner A.
2. While the drain remains active, make A lose the daemon-task lease long enough for B to claim the task and start a consumer.
3. Make B lose its lease long enough for A to reclaim the task before the next owner observation made by the DDL.
4. Return A from the owner lookup at both observable boundaries.
5. Verify that the current UUID-set convergence can complete without having fenced B, despite B having admitted a consumer.
The normal daemon heartbeat timeout makes this an uncommon failover-plus-long-DDL scenario, not the fast startup race from #26181.
### Additional information
Related PR discussion: #26183.
Likely design direction, to be evaluated rather than implemented blindly:
- add a durable daemon ownership epoch or durable per-job DDL fence;
- bind fence installation, task assignment, and consumer admission to that epoch/token;
- make cleanup and renewal conditional on the same token so delayed messages cannot revive or remove another generation fence;
- add an A -> B -> A handoff test and a post-final-read handoff test.
This is a cross-taskservice/ISCP protocol change and should be designed independently of the scoped readiness fix in #26183.
Contributor guide
Assessment
This issue has not been assessed yet.