google-gemini / google-gemini/gemini-cli

v0.55.1: `--model gemini-3.1-pro-preview` silently served by a 2.5-series model under oauth-personal auth

Open
#28,825 2 comments 1 reaction 0 assignees View on GitHub
area/agent kind/bug priority/p2 status/bot-triaged
Dominant language
TypeScript
Stars
107k
Forks
14.6k
Avg merge
2d 3h
Merged PRs (30d)
45

Description

Requesting `gemini-3.1-pro-preview` with "Login with Google" credentials that lack 3.1 entitlement returns a **successful response from a different model**, with no error, no warning, and no indication in the CLI output that substitution occurred.

This is adjacent to #26938 (closed, v0.41.2, aux-role leakage) but distinct: this is entitlement-driven substitution of the *primary* model on the current release, and the substitute target is not stable.

## Repro

```bash
mkdir -p ~/.gemini
printf %s '{"security":{"auth":{"selectedType":"oauth-personal"}}}' > ~/.gemini/settings.json
# ~/.gemini/oauth_creds.json = Login-with-Google credentials without 3.1 entitlement
export GOOGLE_CLOUD_PROJECT=
gemini --yolo --skip-trust --model=gemini-3.1-pro-preview --prompt="Reply with exactly: PONG"
```

Response arrives normally (`PONG`). The session log records what actually served it:

```
~/.gemini/tmp/*/chats/session-*.jsonl
{"type":"gemini","content":"PONG","tokens":{...},"model":"gemini-2.5-flash"}
```

Requested `gemini-3.1-pro-preview`, served `gemini-2.5-flash` — two generations down.

## The substitute is not consistent

Same credentials, same model flag, two environments:

| Environment | Requested | Actually served |
|---|---|---|
| Docker container (harbor agent runner) | `gemini-3.1-pro-preview` | `gemini-2.5-pro` |
| Host shell, `GOOGLE_CLOUD_PROJECT` set | `gemini-3.1-pro-preview` | `gemini-2.5-flash` |

So it isn't a fixed documented fallback that a caller could account for — what you receive varies with context.

## Things ruled out

- **Wrong model ID** — `gemini-3.1-pro-preview` is current per the [API docs](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview), and the bundle contains 57 references to it, so the CLI knows the model.
- **Stale CLI** — reproduced on 0.55.1 (`@latest`).
- **Missing project** — reproduced with `GOOGLE_CLOUD_PROJECT` set.
- **Preview-features setting** — `{"previewFeatures": true}` and `{"general":{"previewFeatures":true}}` both still yield 2.5-flash.

## Why this matters beyond cost control

We hit this while running a benchmark comparison across three model families. The harness requested 3.1 Pro; every trial silently executed on a 2.5-series model and produced normal-looking scores. Had we not recorded the observed model per trial, we would have published results labelled "Gemini 3.1 Pro" that were measured on a different, older model.

Any published benchmark number produced through the CLI on personal credentials is exposed to this, and nothing in the CLI's own output would reveal it.

## Requests

1. **Error rather than substitute.** If the requested model is unavailable to the credential, fail with a clear message.
2. **Failing that, warn on stderr** whenever the served model differs from the requested one.
3. **A `--strict-model` flag** (proposed in #26938) that exits non-zero rather than falling back. For automated and benchmarking use this is the difference between a usable tool and an unusable one.

At minimum, the served model should be surfaced somewhere the caller can see without parsing `~/.gemini/tmp` session logs after the fact.

## Environment

- `@google/gemini-cli` 0.55.1, Node 22.11.0, Ubuntu 22.04 x86_64
- Auth: `oauth-personal` (Login with Google), account without 3.1 entitlement
- Also reproduced inside a Debian-based Docker container via the same auth path

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the `gemini --model=gemini-3.1-pro-preview` command with the `oauth-personal` settings in `~/.gemini/settings.json`, then compare the requested model with the `model` recorded in `~/.gemini/tmp/*/chats/session-*.jsonl`. Trace the CLI's model and entitlement handling until the substitution is identified; done means an unavailable requested model is surfaced clearly instead of silently succeeding.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
authentication, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.