pingcap / pingcap/ticdc

flaky test: ddl_for_split_tables_with_merge_and_split fails after TiCDC capture suicide

Open
#5,176 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Failure

PR: https://github.com/pingcap/ticdc/pull/5098
CI: https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_mysql_integration_heavy/2058/display/redirect
Job: pull_cdc_mysql_integration_heavy #2058
Failed group: G05
Case: ddl_for_split_tables_with_merge_and_split with mysql sink

Evidence

Jenkins failed in the Test stage for TEST_GROUP = 'G05' with script returned exit code 1.

The failing command was the split-table helper path. merge_table_with_retry repeatedly failed for table 119:

{ "success": false, "error": "Can't not find maintainer for changefeed: test" }
{ "success": false, "error": "[CDC:ErrTableIsNotFounded]table is not found%!(EXTRA string=tableID, int64=119)" }
merge table 119 failed after 10 retries

The case logs show both TiCDC captures exited before the helper finished:

Error: [CDC:ErrCaptureSuicide]capture suicide

cdc0.log shows the direct cause as etcd session loss:

[WARN] [etcd_watcher.go:70] ["session is disconnected"] [error="[CDC:ErrEtcdSessionDone]the etcd session is done"]
[ERROR] [server.go:152] ["cdc server exits with error"] [error="[CDC:ErrCaptureSuicide]capture suicide"]

cdc1.log exits with the same ErrCaptureSuicide. Around the same time, down_pd.log shows etcd/PD instability: slow linearizable reads, ReadIndex retry, TSO save timestamp failure, not leader, and slow fsync.

I did not find a table-route conflict / RouteAdmin error in the captured CDC logs. The observed failure is that the CDC cluster lost its maintainer after both captures committed suicide, then the test helper kept calling merge-table.

Expected

The test should not fail as a table scheduling failure when the underlying CDC captures have already exited due to PD/etcd session loss. It should either tolerate transient maintainer unavailability with a meaningful wait/retry path, or report the capture-suicide root cause directly.

Notes

This looks like a flaky integration-test/environment failure rather than a regression in the table-route conflict detector branch. The failure URL above should be kept as the reproduction evidence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the G05 case ddl_for_split_tables_with_merge_and_split and its merge_table_with_retry helper, then review the linked CI logs and cdc0.log/cdc1.log around ErrCaptureSuicide. Determine how the test handles maintainer loss after etcd session failure; done means it reports the capture-suicide cause or handles the transient wait without misreporting a table-scheduling failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.