matrixorigin / matrixorigin/matrixone

[Bug]: ISCP DDL drain lacks an ownership-generation quiescence proof

Open
#26,188 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

`main` / follow-up to #26181 and #26183

### Commit ID

`ff0a3faf2b4c245b4682abf019a99ccb63e62568`

### Other Environment Information

- Deployment: Multi-CN with the ISCP daemon task able to fail over between CNs.
- Impact status: protocol-level correctness gap. It is not the assignment-before-executor-readiness failure reported in #26181, and no separate production incident is claimed here.

### Actual Behavior

`drainIndexCdcTaskConsumer` fences only the daemon-task owners that it observes. Its convergence condition accepts a fresh read of a previously fenced CN UUID.

This is not an ownership-generation linearization point. During a long DDL, ownership can transiently move from CN A to CN B and back to CN A between two owner reads, or move after the final read. CN B can start an ISCP consumer without ever receiving a drain request. The DDL can then complete with no proof that every consumer generation which ran during the operation was quiesced.

The current implementation improves the common readiness and observed-handoff paths: it retries startup readiness, re-reads owners, fences observed CNs, renews fences, and carries fences across executor replacement in one CN process. It intentionally does not provide a durable cross-process ownership proof.

### Expected Behavior

An index/CDC-affecting DDL must either:

1. obtain a quiescence proof tied to a stable daemon-task ownership generation; or
2. fail/retry when ownership changes before that proof is established.

A consumer must reject admission when its ownership generation is fenced, including after executor or CN restart. Stale renew/remove messages must not affect a newer generation.

### Steps to Reproduce

A deterministic integration or fault-injection test should model the following sequence:

1. Start a long ISCP-affecting DDL and let it observe and drain owner A.
2. While the drain remains active, make A lose the daemon-task lease long enough for B to claim the task and start a consumer.
3. Make B lose its lease long enough for A to reclaim the task before the next owner observation made by the DDL.
4. Return A from the owner lookup at both observable boundaries.
5. Verify that the current UUID-set convergence can complete without having fenced B, despite B having admitted a consumer.

The normal daemon heartbeat timeout makes this an uncommon failover-plus-long-DDL scenario, not the fast startup race from #26181.

### Additional information

Related PR discussion: #26183.

Likely design direction, to be evaluated rather than implemented blindly:

- add a durable daemon ownership epoch or durable per-job DDL fence;
- bind fence installation, task assignment, and consumer admission to that epoch/token;
- make cleanup and renewal conditional on the same token so delayed messages cannot revive or remove another generation fence;
- add an A -> B -> A handoff test and a post-final-read handoff test.

This is a cross-taskservice/ISCP protocol change and should be designed independently of the scoped readiness fix in #26183.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.