pingcap / pingcap/tiup

When cdc server cannot connected, tiup stop cdc takes more than 20minutes

Open
#1,241 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/investigating type/bug
Dominant language
Go
Stars
466
Forks
338
Avg merge
3d 7h
Merged PRs (30d)
8

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
  • CDC status is down, execute "$ tiup cluster restart tidb-test"
  1. What did you expect to see?
    If cdc server network cannot be connected, return error in less then 5minutes.

  2. What did you see instead?

  • return error more than 20minutes.

  • tiup log:

2021-03-24T07:58:59.614Z INFO Execute command {"command": "tiup cluster restart tidb-test"}

2021-03-24T08:01:03.055Z INFO Stop cdc 34.217.25.239:8300 success
2021-03-24T08:02:06.352Z ERROR SSHCommand {"host": "52.38.214.137", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c "systemctl daemon-reload && systemctl stop cdc-8300.service"", "error": "ssh: handshake failed: EOF", "stdout": "", "stderr": ""}
2021-03-24T08:24:26.668Z INFO SSHCommand {"host": "18.236.187.81", "port": "22", "cmd": "export LANG=C; PATH=$PATH:/usr/bin:/usr/sbin sudo -H bash -c "systemctl daemon-reload && systemctl stop cdc-8300.service"", "stdout": "", "stderr": ""}
2021-03-24T08:24:26.668Z INFO Stop cdc 18.236.187.81:8300 success
2021-03-24T08:24:26.668Z DEBUG TaskFinish {"task": "RestartCluster", "error": "failed to stop: failed to stop cdc: failed to stop: cdc 52.38.214.137:8300,

  1. What version of TiUP are you using (tiup --version)?
    $ tiup --version
    v1.3.6 tiup
    Go Version: go1.13
    Git Branch: release-1.3
    GitHash: ba978b1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing tiup cluster restart tidb-test with a CDC host that cannot be reached, then trace the SSHCommand handling shown in the logs. Done means the restart reports the CDC stop failure within five minutes instead of taking more than 20 minutes, while still reporting the unreachable host.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, devops, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.