Flaky end-to-end Upgrade Test: MoveTables steps fail with tablet dial timeouts or a lost topo lock lease

The end-to-end Upgrade Test job is flaky on main. It failed on 3 of the last 9 main runs (2026-08-28 168d0fd, 2026-09-10 9fec2db, 2026-09-18 5221b76), and it failed twice in a row on #836, whose diff does not touch the operator binary at all (Makefile, go.mod/go.sum tool deps, build-tag-gated tools/tools.go).

All failures land in the MoveTables portion of test/endtoend/upgrade_test.sh, right after the upgrade has rolled the vttablet pods. Two signatures so far.

Signature 1: SwitchTraffic cannot refresh a tablet (dial i/o timeout)

Seen on main 9fec2db (run 34493911915) and on #836 attempts 1 and 2 (run 35413383235):

SwitchTraffic for rdonly
W... keyspace.go:92] RefreshTabletsByShard: failed to refresh zone1-2548885007: Code: DEADLINE_EXCEEDED
 Error: rpc error: code = Unknown desc = cannot switch traffic for workflow commerce2customer at this time: could not refresh all of the tablets involved in the operation:
failed to successfully refresh all tablets in the commerce/- source shard (<nil>):
  failed to refresh tablet zone1-2548885007: Code: DEADLINE_EXCEEDED
rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing: dial tcp 10.244.0.21:15999: i/o timeout"
SwitchTraffic for rdonly and replica failed

Only the tablet alias and pod IP differ between occurrences. vtctld's RefreshState RPC to one vttablet's gRPC port (15999) times out at the 30s deadline, so the tablet record in topo is pointing at a pod endpoint that is not reachable at that moment.

Signature 2: MoveTables Complete loses its topo lock lease

Seen on main 5221b76 (run 35394202078). SwitchTraffic for rdonly/replica and for primary both succeed, then:

Complete MoveTables
W... topotools.RefreshTabletsByShard ...
{"logger":"etcd-client","caller":"v3@v3.7.1/retry_interceptor.go:68","msg":"retrying of unary invoker failed", ...}
W... topo.(*Server).internalLock.func1 ...
 Error: rpc error: code = Unknown desc = node doesn't exist: lease
MoveTables Complete failed

The same tablet-refresh warnings are firing, and vtctld's etcd lock lease disappears while it holds the keyspace lock for Complete.

Why the test is exposed to this

upgrade_test.sh gates the MoveTables steps only on checkPodStatusWithTimeout "example-vttablet-zone1(.*)3/3(.*)Running(.*)" 6 and waitForKeyspaceToBeServing, then runs Create, SwitchTraffic (rdonly/replica), SwitchTraffic (primary), and Complete with fixed sleep 10s between them. Nothing verifies that every tablet's gRPC endpoint is reachable and its topo record current after the rolling restart, and none of the steps retry on DEADLINE_EXCEEDED.

Suggested follow-ups

• Before the MoveTables steps, wait until every tablet in the affected keyspaces is reachable, e.g. loop until vtctldclient GetTablets plus a per-tablet health or RefreshState call succeeds for all of them, instead of relying on pod Running and a fixed sleep.
• Retry SwitchTraffic and Complete a bounded number of times on DEADLINE_EXCEEDED / "could not refresh all of the tablets", since a tablet that is mid-restart is expected during an upgrade.
• Look into the node doesn't exist: lease failure separately: capture the vtctld and etcd logs on failure, and check whether the lock lease TTL is too short for a CPU-starved runner while the tablets are restarting.

Start with test/endtoend/upgrade_test.sh and trace the MoveTables sequence after the vttablet rolling restart, including its pod-status checks and fixed sleeps. Review the cited Upgrade Test runs, then capture vtctld and etcd logs while checking tablet reachability and lease behavior. Done means the test waits for usable tablet endpoints and handles the identified transient failures without masking persistent errors.