flaky test: ddl_for_split_tables_with_merge_and_split fails after TiCDC capture suicide
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Failure
PR: https://github.com/pingcap/ticdc/pull/5098
CI: https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_mysql_integration_heavy/2058/display/redirect
Job: pull_cdc_mysql_integration_heavy #2058
Failed group: G05
Case: ddl_for_split_tables_with_merge_and_split with mysql sink
Evidence
Jenkins failed in the Test stage for TEST_GROUP = 'G05' with script returned exit code 1.
The failing command was the split-table helper path. merge_table_with_retry repeatedly failed for table 119:
{ "success": false, "error": "Can't not find maintainer for changefeed: test" }
{ "success": false, "error": "[CDC:ErrTableIsNotFounded]table is not found%!(EXTRA string=tableID, int64=119)" }
merge table 119 failed after 10 retries
The case logs show both TiCDC captures exited before the helper finished:
Error: [CDC:ErrCaptureSuicide]capture suicide
cdc0.log shows the direct cause as etcd session loss:
[WARN] [etcd_watcher.go:70] ["session is disconnected"] [error="[CDC:ErrEtcdSessionDone]the etcd session is done"]
[ERROR] [server.go:152] ["cdc server exits with error"] [error="[CDC:ErrCaptureSuicide]capture suicide"]
cdc1.log exits with the same ErrCaptureSuicide. Around the same time, down_pd.log shows etcd/PD instability: slow linearizable reads, ReadIndex retry, TSO save timestamp failure, not leader, and slow fsync.
I did not find a table-route conflict / RouteAdmin error in the captured CDC logs. The observed failure is that the CDC cluster lost its maintainer after both captures committed suicide, then the test helper kept calling merge-table.
Expected
The test should not fail as a table scheduling failure when the underlying CDC captures have already exited due to PD/etcd session loss. It should either tolerate transient maintainer unavailability with a meaningful wait/retry path, or report the capture-suicide root cause directly.
Notes
This looks like a flaky integration-test/environment failure rather than a regression in the table-route conflict detector branch. The failure URL above should be kept as the reproduction evidence.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the G05 case ddl_for_split_tables_with_merge_and_split and its merge_table_with_retry helper, then review the linked CI logs and cdc0.log/cdc1.log around ErrCaptureSuicide. Determine how the test handles maintainer loss after etcd session failure; done means it reports the capture-suicide cause or handles the transient wait without misreporting a table-scheduling failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100