npm bin/copilot is a loader, not a version pin: same path served 1.0.77 then 1.0.78 101s apart; --prefer-version works but is undocumented
- Lingua principale
- Shell
- Stelle
- 11.2k
- Fork
- 1.9k
- Merge medio
- 14h 16m
- PR unite (30g)
- 6
Descrizione
### Describe the bug
The globally-installed npm shim — `$(npm prefix -g)/bin/copilot` — is a **loader, not a version pin**. Two invocations of the same path, 101 seconds apart with nothing touched in between, ran two different CLI versions (1.0.77, then 1.0.78), while the installed npm packages continued to declare `1.0.77`. Nothing an offline check can inspect changed.
This is fine for interactive use, and I'm not asking for auto-update to stop being the default. The problem is for **reproducible evals and benchmarks**: pointing a harness at that path is the natural way to "pin the CLI," and it silently isn't one. A run can straddle two CLI versions, and every result stamped "pinned 1.0.77" becomes unverifiable after the fact.
There is a mechanism that *does* pin — `--prefer-version ` — but it is not in `--help` and I only found it by reading the loader. Meanwhile the two knobs that *are* documented (`--no-auto-update`, `COPILOT_AUTO_UPDATE=false`) do **not** freeze the version to the installed one (details below).
**The good part, stated up front:** the CLI already emits `copilotVersion` on `session.start`. That is the only reason we caught this — our harness compares observed-vs-pinned per trial and quarantined the run after one request instead of twelve. A harness that checks `copilot --version` once at startup would not have noticed. Please keep that field.
### Affected version
`1.0.77` → `1.0.78` (the swap itself). macOS 15 / arm64, node v24.18.1, npm global-prefix install.
### Steps to reproduce the behavior
Observed live, then re-confirmed today from the same machine.
**1. The original observation (2026-08-05, macOS/arm64).** With `COPILOT_CLI_PATH="$(npm prefix -g)/bin/copilot"`, `copilot --version` read `1.0.77`, and a session started immediately after recorded:
```
{"type":"session.start","data":{"sessionId":"06959cca-…","copilotVersion":"1.0.77",
"startTime":"2026-08-05T01:39:53.221Z", …}}
```
101 seconds later, the **same path**, same shell, nothing installed or changed in between:
```
2026-08-05T01:41:34.180Z [INFO] Starting Copilot CLI: 1.0.78
2026-08-05T01:41:34.784Z [INFO] No update needed, current version is 1.0.78, fetched latest release is v1.0.78
2026-08-05T01:41:43.170Z [DEBUG] … User-agent: copilot/1.0.78 (client/github/cli darwin v24.18.1) …
```
A `1.0.78/` directory appeared in `~/Library/Caches/copilot/pkg/darwin-arm64/` at `01:39:56Z` — three seconds after that `session.start`, i.e. **during** the first session. Throughout, both `@github/copilot/package.json` and `@github/copilot-darwin-arm64/package.json` still read `1.0.77`.
**2. Current state on the same machine** — the installed npm packages still declare 1.0.77, and the pkg cache holds three builds side by side:
```console
$ node -p "require('.../node_modules/@github/copilot/package.json').version"
1.0.77
$ node -p "require('.../@github/copilot/node_modules/@github/copilot-darwin-arm64/package.json').version"
1.0.77
$ ls ~/Library/Caches/copilot/pkg/darwin-arm64/
1.0.64-0 1.0.77 1.0.78
```
**3. What each knob actually does** (all four run just now, back to back, same shell; the pkg cache was unchanged before and after):
```console
$ copilot --version
GitHub Copilot CLI 1.0.78. # installed package says 1.0.77
$ copilot --no-auto-update --version
GitHub Copilot CLI 1.0.78. # documented flag — does NOT pin to the installed build
$ COPILOT_AUTO_UPDATE=false copilot --version
GitHub Copilot CLI 1.0.78. # documented env var — same
$ CI=true copilot --version
GitHub Copilot CLI 1.0.78. # CI default (auto-update off) — same
$ copilot --prefer-version 1.0.77 --version
GitHub Copilot CLI 1.0.77. # this one works — and is not in --help
```
`--help` lists `--no-auto-update` ("Disable downloading CLI update automatically…"). It does not mention `--prefer-version`.
### Expected behavior
Three things, in priority order. None of them is "stop auto-updating."
1. **Document that the npm `bin/copilot` path does not pin a version.** It resolves to the newest build in the shared `pkg` cache, which can change between two consecutive invocations. Anyone wiring a harness, benchmark, or eval to that path is reasonably assuming otherwise.
2. **Document `--prefer-version ` as the supported pin** (or provide a supported env-var equivalent for tools that build an argv they don't fully control). It already does exactly the right thing — it's just unfindable. This is the smallest possible fix and it closes most of the gap.
3. **Clarify or fix `--no-auto-update` / `COPILOT_AUTO_UPDATE=false`.** Reading the loader source, `--no-auto-update`, `--prefer-version`, and `COPILOT_AUTO_UPDATE=false` all clear the same "auto-update enabled" predicate that guards the redirect-to-newest-cached-build branch, so I expected `--no-auto-update` to run the installed 1.0.77. It ran 1.0.78. I can't tell from outside whether that's a bug or whether the flag is scoped strictly to "don't download" while the cache redirect is separate — the help text is consistent with the narrower reading. Either way, a user who reaches for `--no-auto-update` to stabilize a version will be misled, so it's worth stating explicitly which it is.
Keep emitting `copilotVersion` on `session.start` — per-session observability is what makes this detectable after the fact, and it's the difference between a quarantined run and twelve silently-invalid results.
### Additional context
- **OS / arch:** macOS 15, arm64 (Apple Silicon). **Node:** v24.18.1 (harness) / v22.22.3 (shell). **Shell:** zsh. **Install:** `npm install -g @github/copilot` into a non-default global prefix.
- Everything above is what was observed on this one machine with these two versions. I haven't tried to reproduce on Linux or Windows, and I'm not claiming anything about the update trigger's timing beyond the timestamps shown.
- Scope note for fairness: this only bites non-interactive/automated use. For an interactive terminal session, auto-updating to the newest cached build is a sensible default.
- Detection cost us 1 premium request instead of 12, entirely because of the `session.start.copilotVersion` field.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with the npm bin/copilot loader and its --help output, then compare the documented --no-auto-update behavior with --prefer-version and the shared pkg cache described in the report. Done means the supported version-pinning behavior and the scope of auto-update controls are clear to users, while session.start continues to emit copilotVersion.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- node.js, shell
- Ambito
- cli, documentation
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 68/100