Agent host SSH: 'Update Server' is a silent no-op (or downgrades) while update API lags the desktop rollout — protocol-version deadlock with no recovery path
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
Type: Bug
### Summary
When the desktop client is on version N+1 (via desktop auto-update) but `update.code.visualstudio.com/api/latest` still points at version N, a remote agent host over SSH becomes **permanently unusable**, and the offered **"Update Server" button silently does nothing** (or actively downgrades). There is no recovery path in the product; users are stuck with the incompatibility banner until Microsoft flips the update-API pointer.
This is #322068 again, but with the root cause traced: that issue was closed as fixed, yet the underlying deadlock is structural and just reproduced for days on 1.128.0.
### Concrete timeline (reproduced repeatedly, 2026-07-11 → 2026-07-13)
- Desktop auto-updated to **1.128.0** (`fc3def67…`), which speaks AHP **0.5.1**.
- `https://update.code.visualstudio.com/api/latest/server-linux-x64/stable` still returns **1.127.0** (`4fe60c8b…`, timestamp 06-30) — six days after 1.128.0 shipped. All quality segments (`darwin-arm64`, `cli-alpine-x64`) agree, so this is the pointer's normal staged-rollout lag, not a CDN glitch.
- The supervisor (`code … agent host`) on the remote resolves "latest" via that API (`UpdateService::get_latest_commit`, `cli/src/tunnels/agent_host.rs`), so it spawns a **1.127** server (AHP **0.5.0**).
- Client handshake fails: `Client offered protocol versions [0.5.1], none of which are compatible with this server's version 0.5.0 (server accepts ^0.5.0)`.
- The error carries `_meta.vscodeUpgradeMethod = "_vscodeUpgrade"`, so the banner offers **Update Server** (#316750 / #322305). Clicking it → `POST /upgrade` on the management socket → supervisor asks the same update API → `running_commit == latest_commit` → `upgradeNeeded: false` → **no-op**. UI surfaces nothing actionable (just a generic error toast).
- Worse: if the remote is manually seeded with a 1.128 server, the supervisor's periodic update loop (or a triggered upgrade) can **downgrade it back to 1.127**, because the upgrade check is `running != latest`, not `running < latest` — observed live: a manual `POST /upgrade` "upgraded" a 1.128 server down to 1.127.
### Why this is structural
Three components disagree about what "latest" means:
1. **Desktop** updates via its own staged channel (ShipIt et al.) and can be on N+1 early.
2. **AHP** requires caret-compatible protocol versions, and 1.127→1.128 bumped 0.5.0→0.5.1 with the client offering only `[0.5.1]`.
3. **The supervisor** trusts `api/latest`, which lags desktop rollout by days.
Any minor release that bumps the AHP protocol version reopens this window for every user whose desktop updates before the API pointer flips. During the window, "Update Server" is a dead button and remote agent-host sessions are unusable.
### Suggested fixes (any one of these breaks the deadlock)
1. **Client-driven target version**: the client knows its own commit; let `_vscodeUpgrade` pass the desired commit to the supervisor's `/upgrade` endpoint, which already supports commit-pinned downloads (`VSCODE_CLI_INITIAL_AH_VERSION` proves the plumbing exists — it is documented as a test hook for exactly this flow).
2. **Fall back across the lockfile**: when negotiation fails and the API's latest equals the running (incompatible) commit, surface an honest message ("server build not yet published for your client — retry later") instead of a no-op Update button.
3. **Client-side protocol tolerance**: offer `[0.5.1, 0.5.0]` when the client can degrade gracefully, mirroring the lenient negotiation added server-side in #316750.
### Workaround we're using (for other affected users)
Wrap the remote CLI binary to pin the server commit and blackhole the update check:
```bash
mv ~/.vscode-server/code- ~/.vscode-server/code--real
printf '#!/bin/bash\nexport VSCODE_CLI_INITIAL_AH_VERSION=\nexport VSCODE_CLI_UPDATE_URL=http://127.0.0.1:1\nexec "$0-real" "$@"\n' > ~/.vscode-server/code-
chmod +x ~/.vscode-server/code-
# kill existing `agent host` supervisors; the client respawns one through the wrapper
```
(`VSCODE_CLI_UPDATE_URL` blackholing is safe: `get_latest_release` falls back to the cached release on update-check failure.)
### Environment
- Desktop: VS Code stable 1.128.0 (`fc3def67…`), macOS
- Remote: Amazon Linux 2023 x86_64 over SSH ("remote agents" / cdd tunnel), supervisor from `code-fc3def67… agent host`
- Reproduced continuously 2026-07-11 → 2026-07-13 while `api/latest/server-linux-x64/stable` = 1.127.0
Contributor guide
Research direction
Read cli/src/tunnels/agent_host.rs, especially UpdateService::get_latest_commit, then trace the management socket POST /upgrade flow. Done means the Update Server path has an actionable outcome when api/latest lags and cannot replace a newer running server with an older one; validate the client-driven, fallback, or protocol-tolerance behavior described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, devtools, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100