google-gemini / google-gemini/gemini-cli
v0.55.1: `--model gemini-3.1-pro-preview` silently served by a 2.5-series model under oauth-personal auth
- Dominant language
- TypeScript
- Stars
- 107k
- Forks
- 14.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 45
Description
Requesting `gemini-3.1-pro-preview` with "Login with Google" credentials that lack 3.1 entitlement returns a **successful response from a different model**, with no error, no warning, and no indication in the CLI output that substitution occurred.
This is adjacent to #26938 (closed, v0.41.2, aux-role leakage) but distinct: this is entitlement-driven substitution of the *primary* model on the current release, and the substitute target is not stable.
## Repro
```bash
mkdir -p ~/.gemini
printf %s '{"security":{"auth":{"selectedType":"oauth-personal"}}}' > ~/.gemini/settings.json
# ~/.gemini/oauth_creds.json = Login-with-Google credentials without 3.1 entitlement
export GOOGLE_CLOUD_PROJECT=
gemini --yolo --skip-trust --model=gemini-3.1-pro-preview --prompt="Reply with exactly: PONG"
```
Response arrives normally (`PONG`). The session log records what actually served it:
```
~/.gemini/tmp/*/chats/session-*.jsonl
{"type":"gemini","content":"PONG","tokens":{...},"model":"gemini-2.5-flash"}
```
Requested `gemini-3.1-pro-preview`, served `gemini-2.5-flash` — two generations down.
## The substitute is not consistent
Same credentials, same model flag, two environments:
| Environment | Requested | Actually served |
|---|---|---|
| Docker container (harbor agent runner) | `gemini-3.1-pro-preview` | `gemini-2.5-pro` |
| Host shell, `GOOGLE_CLOUD_PROJECT` set | `gemini-3.1-pro-preview` | `gemini-2.5-flash` |
So it isn't a fixed documented fallback that a caller could account for — what you receive varies with context.
## Things ruled out
- **Wrong model ID** — `gemini-3.1-pro-preview` is current per the [API docs](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview), and the bundle contains 57 references to it, so the CLI knows the model.
- **Stale CLI** — reproduced on 0.55.1 (`@latest`).
- **Missing project** — reproduced with `GOOGLE_CLOUD_PROJECT` set.
- **Preview-features setting** — `{"previewFeatures": true}` and `{"general":{"previewFeatures":true}}` both still yield 2.5-flash.
## Why this matters beyond cost control
We hit this while running a benchmark comparison across three model families. The harness requested 3.1 Pro; every trial silently executed on a 2.5-series model and produced normal-looking scores. Had we not recorded the observed model per trial, we would have published results labelled "Gemini 3.1 Pro" that were measured on a different, older model.
Any published benchmark number produced through the CLI on personal credentials is exposed to this, and nothing in the CLI's own output would reveal it.
## Requests
1. **Error rather than substitute.** If the requested model is unavailable to the credential, fail with a clear message.
2. **Failing that, warn on stderr** whenever the served model differs from the requested one.
3. **A `--strict-model` flag** (proposed in #26938) that exits non-zero rather than falling back. For automated and benchmarking use this is the difference between a usable tool and an unusable one.
At minimum, the served model should be surfaced somewhere the caller can see without parsing `~/.gemini/tmp` session logs after the fact.
## Environment
- `@google/gemini-cli` 0.55.1, Node 22.11.0, Ubuntu 22.04 x86_64
- Auth: `oauth-personal` (Login with Google), account without 3.1 entitlement
- Also reproduced inside a Debian-based Docker container via the same auth path
Contributor guide
Research direction
Start by reproducing the `gemini --model=gemini-3.1-pro-preview` command with the `oauth-personal` settings in `~/.gemini/settings.json`, then compare the requested model with the `model` recorded in `~/.gemini/tmp/*/chats/session-*.jsonl`. Trace the CLI's model and entitlement handling until the substitution is identified; done means an unavailable requested model is surfaced clearly instead of silently succeeding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- authentication, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100