matrixorigin / matrixorigin/matrixone

[Bug]: RESUME CDC TASK cannot clear a target-table error while task state is Running

Open
#26,633 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

# Summary

`RESUME CDC TASK ` cannot recover a table-level CDC error when `SHOW CDC TASK` reports the task state as `running`.

This conflicts with the CDC guide's documented recovery flow:

- TC-501 says that a missing target table records an error while the task remains running.
- TC-502 says that, after the target is fixed, `RESUME CDC TASK` clears the error and synchronization resumes.

After restoring the target table, the regular `RESUME CDC TASK` command is rejected because the task is still `Running`. The table error remains and replication cannot continue through the documented command. The restart variant does recover, which isolates the issue to the normal resume/state transition path.

## Version

- MatrixOne `main`: `820ee12144439c5b63c03b261f4330ec0364fdb3`
- Isolated two-CN MatrixOne deployment
- Three independent reproductions

## Minimal reproduction

1. Create matching source and target tables and create a table-level CDC task with `NoFull=true`.
2. Wait until `SHOW CDC TASK` reports a checkpoint for the source table.
3. Verify one post-checkpoint source insert reaches the target.
4. Drop the target table, then insert another source row.
5. Wait for `SHOW CDC TASK` to expose a non-empty `err_msg` for the missing target table. The task state is `running`, as documented by TC-501.
6. Recreate the same target table.
7. Run:

```sql
RESUME CDC TASK cdc_lifecycle;
```

## Actual result

The command is rejected:

```text
internal error: Task cdc_lifecycle status can not be change, now it is Running
```

The table error remains in `SHOW CDC TASK` and the CDC task does not retry the restored target through the regular resume command.

As a control, the following command eventually clears the error and delivers both the row written while the target was absent and a later row:

```sql
RESUME CDC TASK cdc_lifecycle 'restart';
```

## Expected result

After the target table has been restored, the documented command:

```sql
RESUME CDC TASK cdc_lifecycle;
```

should clear the table error and resume processing from the recorded watermark, even though the task's top-level state remains `running` during the per-table error.

Alternatively, if a restart-only recovery model is intentional, the command should not be documented as the TC-502 recovery path and the SQL error should direct users to the required restart form.

## Reproduction evidence

All three runs observed the same sequence:

1. Missing target table produced a visible table-level error.
2. Normal `RESUME CDC TASK` was rejected with the task-state error above.
3. The error remained visible after the rejected command.
4. The restart form cleared the error and delivered the pending and subsequent source rows.

The earlier pause/resume check in the same isolated deployment also confirmed that ordinary pause recovery does eventually deliver the paused-period change; this issue is specific to a table error while the task reports `running`.

## Related issue

This is related to, but distinct from, #25177. That issue fixed the table-detector path that turns a permanent error into a failed task. This reproduction uses an already-running pipeline and the documented TC-501 behavior where a missing target table keeps the task in `running`; the broken behavior is the incompatibility between that state and the documented TC-502 `RESUME CDC TASK` recovery command.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.