github / github/copilot-cli

npm bin/copilot is a loader, not a version pin: same path served 1.0.77 then 1.0.78 101s apart; --prefer-version works but is undocumented

Aperta
#4,402 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area:installation
Lingua principale
Shell
Stelle
11.2k
Fork
1.9k
Merge medio
14h 16m
PR unite (30g)
6

Descrizione

### Describe the bug

The globally-installed npm shim — `$(npm prefix -g)/bin/copilot` — is a **loader, not a version pin**. Two invocations of the same path, 101 seconds apart with nothing touched in between, ran two different CLI versions (1.0.77, then 1.0.78), while the installed npm packages continued to declare `1.0.77`. Nothing an offline check can inspect changed.

This is fine for interactive use, and I'm not asking for auto-update to stop being the default. The problem is for **reproducible evals and benchmarks**: pointing a harness at that path is the natural way to "pin the CLI," and it silently isn't one. A run can straddle two CLI versions, and every result stamped "pinned 1.0.77" becomes unverifiable after the fact.

There is a mechanism that *does* pin — `--prefer-version ` — but it is not in `--help` and I only found it by reading the loader. Meanwhile the two knobs that *are* documented (`--no-auto-update`, `COPILOT_AUTO_UPDATE=false`) do **not** freeze the version to the installed one (details below).

**The good part, stated up front:** the CLI already emits `copilotVersion` on `session.start`. That is the only reason we caught this — our harness compares observed-vs-pinned per trial and quarantined the run after one request instead of twelve. A harness that checks `copilot --version` once at startup would not have noticed. Please keep that field.

### Affected version

`1.0.77` → `1.0.78` (the swap itself). macOS 15 / arm64, node v24.18.1, npm global-prefix install.

### Steps to reproduce the behavior

Observed live, then re-confirmed today from the same machine.

**1. The original observation (2026-08-05, macOS/arm64).** With `COPILOT_CLI_PATH="$(npm prefix -g)/bin/copilot"`, `copilot --version` read `1.0.77`, and a session started immediately after recorded:

```
{"type":"session.start","data":{"sessionId":"06959cca-…","copilotVersion":"1.0.77",
"startTime":"2026-08-05T01:39:53.221Z", …}}
```

101 seconds later, the **same path**, same shell, nothing installed or changed in between:

```
2026-08-05T01:41:34.180Z [INFO] Starting Copilot CLI: 1.0.78
2026-08-05T01:41:34.784Z [INFO] No update needed, current version is 1.0.78, fetched latest release is v1.0.78
2026-08-05T01:41:43.170Z [DEBUG] … User-agent: copilot/1.0.78 (client/github/cli darwin v24.18.1) …
```

A `1.0.78/` directory appeared in `~/Library/Caches/copilot/pkg/darwin-arm64/` at `01:39:56Z` — three seconds after that `session.start`, i.e. **during** the first session. Throughout, both `@github/copilot/package.json` and `@github/copilot-darwin-arm64/package.json` still read `1.0.77`.

**2. Current state on the same machine** — the installed npm packages still declare 1.0.77, and the pkg cache holds three builds side by side:

```console
$ node -p "require('.../node_modules/@github/copilot/package.json').version"
1.0.77
$ node -p "require('.../@github/copilot/node_modules/@github/copilot-darwin-arm64/package.json').version"
1.0.77
$ ls ~/Library/Caches/copilot/pkg/darwin-arm64/
1.0.64-0 1.0.77 1.0.78
```

**3. What each knob actually does** (all four run just now, back to back, same shell; the pkg cache was unchanged before and after):

```console
$ copilot --version
GitHub Copilot CLI 1.0.78. # installed package says 1.0.77

$ copilot --no-auto-update --version
GitHub Copilot CLI 1.0.78. # documented flag — does NOT pin to the installed build

$ COPILOT_AUTO_UPDATE=false copilot --version
GitHub Copilot CLI 1.0.78. # documented env var — same

$ CI=true copilot --version
GitHub Copilot CLI 1.0.78. # CI default (auto-update off) — same

$ copilot --prefer-version 1.0.77 --version
GitHub Copilot CLI 1.0.77. # this one works — and is not in --help
```

`--help` lists `--no-auto-update` ("Disable downloading CLI update automatically…"). It does not mention `--prefer-version`.

### Expected behavior

Three things, in priority order. None of them is "stop auto-updating."

1. **Document that the npm `bin/copilot` path does not pin a version.** It resolves to the newest build in the shared `pkg` cache, which can change between two consecutive invocations. Anyone wiring a harness, benchmark, or eval to that path is reasonably assuming otherwise.

2. **Document `--prefer-version ` as the supported pin** (or provide a supported env-var equivalent for tools that build an argv they don't fully control). It already does exactly the right thing — it's just unfindable. This is the smallest possible fix and it closes most of the gap.

3. **Clarify or fix `--no-auto-update` / `COPILOT_AUTO_UPDATE=false`.** Reading the loader source, `--no-auto-update`, `--prefer-version`, and `COPILOT_AUTO_UPDATE=false` all clear the same "auto-update enabled" predicate that guards the redirect-to-newest-cached-build branch, so I expected `--no-auto-update` to run the installed 1.0.77. It ran 1.0.78. I can't tell from outside whether that's a bug or whether the flag is scoped strictly to "don't download" while the cache redirect is separate — the help text is consistent with the narrower reading. Either way, a user who reaches for `--no-auto-update` to stabilize a version will be misled, so it's worth stating explicitly which it is.

Keep emitting `copilotVersion` on `session.start` — per-session observability is what makes this detectable after the fact, and it's the difference between a quarantined run and twelve silently-invalid results.

### Additional context

- **OS / arch:** macOS 15, arm64 (Apple Silicon). **Node:** v24.18.1 (harness) / v22.22.3 (shell). **Shell:** zsh. **Install:** `npm install -g @github/copilot` into a non-default global prefix.
- Everything above is what was observed on this one machine with these two versions. I haven't tried to reproduce on Linux or Windows, and I'm not claiming anything about the update trigger's timing beyond the timestamps shown.
- Scope note for fairness: this only bites non-interactive/automated use. For an interactive terminal session, auto-updating to the newest cached build is a sensible default.
- Detection cost us 1 premium request instead of 12, entirely because of the `session.start.copilotVersion` field.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the npm bin/copilot loader and its --help output, then compare the documented --no-auto-update behavior with --prefer-version and the shared pkg cache described in the report. Done means the supported version-pinning behavior and the scope of auto-update controls are clear to users, while session.start continues to emit copilotVersion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
node.js, shell
Ambito
cli, documentation
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
68/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.