pingcap / pingcap/tiup

Cluster upgrade fails when multiple topology hosts resolve to same physical server

Open
#2,730 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

contribution type/bug
Dominant language
Go
Stars
466
Forks
338
Avg merge
3d 7h
Merged PRs (30d)
8

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    A TiDB cluster with the following topology:
  • 3 physical servers with IPs in two subnets: 172.xx.xx.n and 192.168.0.n
  • tiproxy_servers using 192.168.0.n addresses
  • All other components (PD, TiKV, TiDB, CDC, monitoring) using 172.xx.xx.n addresses
  • The 192.168.0.n and 172.xx.xx.n addresses resolve to the same physical machines (e.g., 192.168.0.212 = 172.xx.xx.212)

The monitored components (node_exporter, blackbox_exporter) are configured with a shared deploy_dir: , which is identical across all hosts in the topology.

tiup cluster upgrade <cluster_name> v7.1.9-0.2

  1. What did you expect to see?

Cluster upgrade to complete successfully without errors.

  1. What did you see instead?

Upgrade fails with error:

cannot remove '/tdata1/tidb-deploy/monitored-9100/bin/node_exporter-v1.5.0-d529d2a-linux-amd64.tar.gz',no such file or directory

seems like : TiUP treats the two IP addresses (172.xx.xx.n and 192.168.0.n) as separate hosts. When upgrading, two concurrent tasks deploy node_exporter to the same physical path on the same machine. The first task's command deletes the tar.gz, then the second task's fails with "no such file".

workaround: specify -c 1 to run upgrades serially and bypass this issue.

  1. What version of TiUP are you using (tiup --version)?

v1.16.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the tiup cluster upgrade <cluster_name> v7.1.9-0.2 entry point and reproduce the topology using two addresses that resolve to the same physical servers, comparing parallel execution with -c 1. Trace how monitored components and shared deploy_dir paths are identified during the upgrade; done means the upgrade completes successfully without the missing-tarball error while preserving concurrent upgrades for distinct hosts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.