Cluster upgrade fails when multiple topology hosts resolve to same physical server
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 466
- Forks
- 338
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 8
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
- What did you do?
A TiDB cluster with the following topology:
- 3 physical servers with IPs in two subnets: 172.xx.xx.n and 192.168.0.n
- tiproxy_servers using 192.168.0.n addresses
- All other components (PD, TiKV, TiDB, CDC, monitoring) using 172.xx.xx.n addresses
- The 192.168.0.n and 172.xx.xx.n addresses resolve to the same physical machines (e.g., 192.168.0.212 = 172.xx.xx.212)
The monitored components (node_exporter, blackbox_exporter) are configured with a shared deploy_dir: , which is identical across all hosts in the topology.
tiup cluster upgrade <cluster_name> v7.1.9-0.2
- What did you expect to see?
Cluster upgrade to complete successfully without errors.
- What did you see instead?
Upgrade fails with error:
cannot remove '/tdata1/tidb-deploy/monitored-9100/bin/node_exporter-v1.5.0-d529d2a-linux-amd64.tar.gz',no such file or directory
seems like : TiUP treats the two IP addresses (172.xx.xx.n and 192.168.0.n) as separate hosts. When upgrading, two concurrent tasks deploy node_exporter to the same physical path on the same machine. The first task's command deletes the tar.gz, then the second task's fails with "no such file".
workaround: specify -c 1 to run upgrades serially and bypass this issue.
- What version of TiUP are you using (
tiup --version)?
v1.16.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tiup cluster upgrade <cluster_name> v7.1.9-0.2 entry point and reproduce the topology using two addresses that resolve to the same physical servers, comparing parallel execution with -c 1. Trace how monitored components and shared deploy_dir paths are identified during the upgrade; done means the upgrade completes successfully without the missing-tarball error while preserving concurrent upgrades for distinct hosts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100