github / github/copilot-cli

npm bin/copilot is a loader, not a version pin: same path served 1.0.77 then 1.0.78 101s apart; --prefer-version works but is undocumented

Open
#4,402 0 comments 0 reactions 0 assignees View on GitHub
area:installation
Dominant language
Shell
Stars
11.2k
Forks
1.9k
Avg merge
14h 16m
Merged PRs (30d)
6

Description

### Describe the bug

The globally-installed npm shim — `$(npm prefix -g)/bin/copilot` — is a **loader, not a version pin**. Two invocations of the same path, 101 seconds apart with nothing touched in between, ran two different CLI versions (1.0.77, then 1.0.78), while the installed npm packages continued to declare `1.0.77`. Nothing an offline check can inspect changed.

This is fine for interactive use, and I'm not asking for auto-update to stop being the default. The problem is for **reproducible evals and benchmarks**: pointing a harness at that path is the natural way to "pin the CLI," and it silently isn't one. A run can straddle two CLI versions, and every result stamped "pinned 1.0.77" becomes unverifiable after the fact.

There is a mechanism that *does* pin — `--prefer-version ` — but it is not in `--help` and I only found it by reading the loader. Meanwhile the two knobs that *are* documented (`--no-auto-update`, `COPILOT_AUTO_UPDATE=false`) do **not** freeze the version to the installed one (details below).

**The good part, stated up front:** the CLI already emits `copilotVersion` on `session.start`. That is the only reason we caught this — our harness compares observed-vs-pinned per trial and quarantined the run after one request instead of twelve. A harness that checks `copilot --version` once at startup would not have noticed. Please keep that field.

### Affected version

`1.0.77` → `1.0.78` (the swap itself). macOS 15 / arm64, node v24.18.1, npm global-prefix install.

### Steps to reproduce the behavior

Observed live, then re-confirmed today from the same machine.

**1. The original observation (2026-08-05, macOS/arm64).** With `COPILOT_CLI_PATH="$(npm prefix -g)/bin/copilot"`, `copilot --version` read `1.0.77`, and a session started immediately after recorded:

```
{"type":"session.start","data":{"sessionId":"06959cca-…","copilotVersion":"1.0.77",
"startTime":"2026-08-05T01:39:53.221Z", …}}
```

101 seconds later, the **same path**, same shell, nothing installed or changed in between:

```
2026-08-05T01:41:34.180Z [INFO] Starting Copilot CLI: 1.0.78
2026-08-05T01:41:34.784Z [INFO] No update needed, current version is 1.0.78, fetched latest release is v1.0.78
2026-08-05T01:41:43.170Z [DEBUG] … User-agent: copilot/1.0.78 (client/github/cli darwin v24.18.1) …
```

A `1.0.78/` directory appeared in `~/Library/Caches/copilot/pkg/darwin-arm64/` at `01:39:56Z` — three seconds after that `session.start`, i.e. **during** the first session. Throughout, both `@github/copilot/package.json` and `@github/copilot-darwin-arm64/package.json` still read `1.0.77`.

**2. Current state on the same machine** — the installed npm packages still declare 1.0.77, and the pkg cache holds three builds side by side:

```console
$ node -p "require('.../node_modules/@github/copilot/package.json').version"
1.0.77
$ node -p "require('.../@github/copilot/node_modules/@github/copilot-darwin-arm64/package.json').version"
1.0.77
$ ls ~/Library/Caches/copilot/pkg/darwin-arm64/
1.0.64-0 1.0.77 1.0.78
```

**3. What each knob actually does** (all four run just now, back to back, same shell; the pkg cache was unchanged before and after):

```console
$ copilot --version
GitHub Copilot CLI 1.0.78. # installed package says 1.0.77

$ copilot --no-auto-update --version
GitHub Copilot CLI 1.0.78. # documented flag — does NOT pin to the installed build

$ COPILOT_AUTO_UPDATE=false copilot --version
GitHub Copilot CLI 1.0.78. # documented env var — same

$ CI=true copilot --version
GitHub Copilot CLI 1.0.78. # CI default (auto-update off) — same

$ copilot --prefer-version 1.0.77 --version
GitHub Copilot CLI 1.0.77. # this one works — and is not in --help
```

`--help` lists `--no-auto-update` ("Disable downloading CLI update automatically…"). It does not mention `--prefer-version`.

### Expected behavior

Three things, in priority order. None of them is "stop auto-updating."

1. **Document that the npm `bin/copilot` path does not pin a version.** It resolves to the newest build in the shared `pkg` cache, which can change between two consecutive invocations. Anyone wiring a harness, benchmark, or eval to that path is reasonably assuming otherwise.

2. **Document `--prefer-version ` as the supported pin** (or provide a supported env-var equivalent for tools that build an argv they don't fully control). It already does exactly the right thing — it's just unfindable. This is the smallest possible fix and it closes most of the gap.

3. **Clarify or fix `--no-auto-update` / `COPILOT_AUTO_UPDATE=false`.** Reading the loader source, `--no-auto-update`, `--prefer-version`, and `COPILOT_AUTO_UPDATE=false` all clear the same "auto-update enabled" predicate that guards the redirect-to-newest-cached-build branch, so I expected `--no-auto-update` to run the installed 1.0.77. It ran 1.0.78. I can't tell from outside whether that's a bug or whether the flag is scoped strictly to "don't download" while the cache redirect is separate — the help text is consistent with the narrower reading. Either way, a user who reaches for `--no-auto-update` to stabilize a version will be misled, so it's worth stating explicitly which it is.

Keep emitting `copilotVersion` on `session.start` — per-session observability is what makes this detectable after the fact, and it's the difference between a quarantined run and twelve silently-invalid results.

### Additional context

- **OS / arch:** macOS 15, arm64 (Apple Silicon). **Node:** v24.18.1 (harness) / v22.22.3 (shell). **Shell:** zsh. **Install:** `npm install -g @github/copilot` into a non-default global prefix.
- Everything above is what was observed on this one machine with these two versions. I haven't tried to reproduce on Linux or Windows, and I'm not claiming anything about the update trigger's timing beyond the timestamps shown.
- Scope note for fairness: this only bites non-interactive/automated use. For an interactive terminal session, auto-updating to the newest cached build is a sensible default.
- Detection cost us 1 premium request instead of 12, entirely because of the `session.start.copilotVersion` field.

Contributor guide

Open the contributing guide

Research direction

Start with the npm bin/copilot loader and its --help output, then compare the documented --no-auto-update behavior with --prefer-version and the shared pkg cache described in the report. Done means the supported version-pinning behavior and the scope of auto-update controls are clear to users, while session.start continues to emit copilotVersion.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, shell
Domain
cli, documentation
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.