When cdc server cannot connected, tiup stop cdc takes more than 20minutes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 466
- Forks
- 338
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 8
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
- What did you do?
- CDC status is down, execute "$ tiup cluster restart tidb-test"
-
What did you expect to see?
If cdc server network cannot be connected, return error in less then 5minutes. -
What did you see instead?
-
return error more than 20minutes.
-
tiup log:
2021-03-24T07:58:59.614Z INFO Execute command {"command": "tiup cluster restart tidb-test"}
2021-03-24T08:01:03.055Z INFO Stop cdc 34.217.25.239:8300 success
2021-03-24T08:02:06.352Z ERROR SSHCommand {"host": "52.38.214.137", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c "systemctl daemon-reload && systemctl stop cdc-8300.service"", "error": "ssh: handshake failed: EOF", "stdout": "", "stderr": ""}
2021-03-24T08:24:26.668Z INFO SSHCommand {"host": "18.236.187.81", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c "systemctl daemon-reload && systemctl stop cdc-8300.service"", "stdout": "", "stderr": ""}
2021-03-24T08:24:26.668Z INFO Stop cdc 18.236.187.81:8300 success
2021-03-24T08:24:26.668Z DEBUG TaskFinish {"task": "RestartCluster", "error": "failed to stop: failed to stop cdc: failed to stop: cdc 52.38.214.137:8300,
- What version of TiUP are you using (
tiup --version)?
$ tiup --version
v1.3.6 tiup
Go Version: go1.13
Git Branch: release-1.3
GitHash: ba978b1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing tiup cluster restart tidb-test with a CDC host that cannot be reached, then trace the SSHCommand handling shown in the logs. Done means the restart reports the CDC stop failure within five minutes instead of taking more than 20 minutes, while still reporting the unreachable host.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, devops, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100