`gemini-3.1-flash-live-preview`: a turn left open after an ignored utterance never closes on its own; the caller's next sentence is consumed to close it with zero output tokens

> Re-posted from [`google-gemini/cookbook#1329`](https://github.com/google-gemini/cookbook/issues/1329) at the maintainers' request ([comment](https://github.com/google-gemini/cookbook/issues/1329#issuecomment-5569464227)), so that it reaches the Live engineering team. The report below is unchanged except for the cross-reference to the companion report, which now points to its copy in this repository; the earlier thread is in the original issue.

### Description of the bug

On `gemini-3.1-flash-live-preview` with server-side (automatic) activity detection, the server
sometimes does not react to a caller utterance at all: **no input transcription, no
`interrupted`, no generation**, while the WebSocket is alive and session-resumption updates keep
arriving every ~0.9 s. That half is already reported by several people (links at the bottom).

The part we could not find reported anywhere is what the session does **next**:

1. **The open turn never closes on its own.** In 5 trials, after the ignored utterance the caller
goes silent and we observe the socket for up to **+13.3 s** from the moment the utterance was
swallowed. Nothing arrives in that window: no transcription, no `interrupted`, no
`turnComplete`, no `usageMetadata`. Only `sessionResumptionUpdate`, every ~0.9 s.
2. **What closes it is the caller speaking again.** 5 measurements out of 5: the release arrives
**2.69–2.85 s after the start of the *next* utterance** — which is the normal input
transcription latency for a ~1.8 s phrase on this model — and the turn closes **with zero
output tokens** (`usageMetadata` with no `responseTokenCount`).
3. **Net effect: two caller utterances produce no answer.** The first is ignored; the second is
spent closing the stale turn and gets no generation of its own. If the caller keeps talking,
the cycle can repeat.

We previously believed there was a fixed ~10.3 s "deaf window" after a swallowed utterance
(measured 10.29–10.35 s across 7 of 7 sessions, which looked very much like a server timer).
That was an artefact of our own probe's cadence: it declared the swallow at +6 s and started
speaking again at +7.8 s, and 7.8 + ~2.5 s of transcription latency ≈ 10.3 s. There is no timer.
Measured window lengths were `8.01 / 8.01 / 8.03 / 8.03 / 8.17 s` — with the next utterance always
starting at exactly +5.32 s. The window tracks the caller's cadence, not the clock.

Related, and tested because the documentation suggests it should help: **`audioStreamEnd` does
not flush anything.** 0 reactions in 5 trials, 8 s of silent observation after each flush. No
transcription, no generation, no turn closed, no content recovered. The message is accepted
without error and the connection stays alive. 3 of those trials had the caller leg streaming
continuously (our production shape) and 2 had a full second of real silence around the flush, to
rule out the documented behaviour that sending audio reopens the stream and could cancel the
flush. Same result in both forms.

### Actual vs expected behavior

| | expected | observed |
|---|---|---|
| a turn opened on caller audio that the server does not act on | closes or errors within some bounded time | never closes while the caller is silent; observed to +13.3 s, 5/5 |
| the caller's next utterance | opens its own turn and gets a generation | consumed to close the stale turn, 5/5, with no `responseTokenCount` |
| `audioStreamEnd` in automatic-VAD mode | flushes cached input audio, per the docs | no server reaction at all, 0/5 |

### Environment

| | |
|---|---|
| Model | `gemini-3.1-flash-live-preview` |
| Transport | `google-genai` Python SDK **1.73.1** |
| Endpoint / backend | Gemini API — `generativelanguage.googleapis.com`, API key auth. **Not** Vertex AI. |
| Python / platform | CPython 3.12, macOS arm64 |
| Audio | PCM `s16le`, 16 000 Hz, 1 channel, 16-bit; `audio/pcm;rate=16000`; **20 ms frames sent every 20 ms, continuously, digital zeros during pauses — the sender never stops** |
| Activity detection | server default in the campaigns that measure the production configuration; a pinned `automaticActivityDetection` in the older rate campaign, stated per number below |
| `realtimeInputConfig` | `activityHandling: START_OF_ACTIVITY_INTERRUPTS`, `turnCoverage: TURN_INCLUDES_ONLY_ACTIVITY` (also tested with `TURN_INCLUDES_ALL_INPUT`, no difference — see below) |
| Also configured | `inputAudioTranscription: {}`, `outputAudioTranscription: {}`, `sessionResumption`, `systemInstruction` ~60 000 chars |

Exact UTC timestamps of the sessions cited here are available on request, and we are happy to
provide our Google Cloud project number privately through whichever intake form you prefer — it
is deliberately not in this public body.

### Evidence

Probe timeline of one control trial, verbatim (`t` in seconds from session start; `###` lines are
the probe's own markers, the others are received `LiveServerMessage`s):

```
9.27 ### speak #1 dt_gc=5.92
14.59 ### swallowed start=9.27 detected=14.59
14.59 ### retry #1 (+5.32s)
17.10 ### retry #2 (+7.83s)
17.44 in_tx 'Sì, ci sono.'
17.82 interrupted
17.83 turn_complete
17.83 usage prompt=15695 out=None
```

The utterance at 9.27 s is ignored (nothing between 9.27 and 14.59). The transcription at 17.44 s
arrives **2.85 s after the start of `retry #1`** (14.59) and only **0.34 s after the start of
`retry #2`** (17.10) — far too early to be `retry #2`, so it is the release triggered by
`retry #1`. The turn then closes with `out=None`.

The `audioStreamEnd` arm of the same campaign, for contrast: nothing at all is received between
the flush at 14.33 s and the end of observation at 21.82 s except session-resumption updates.

### How often the state is entered — and why there is no deterministic recipe

We do **not** have a deterministic reproducer for the ignored utterance itself. It is stochastic;
what we have are measured rates, and they move a lot. Stated with the configuration each was
measured under:

| population | configuration | rate |
|---|---|---|
| ~1.8 s utterances, 100 extractions | pinned `automaticActivityDetection` (`prefixPaddingMs=600`, `silenceDurationMs=500`, start `HIGH`, end `LOW`) | **29 %** |
| ~0.6 s utterances, 100 extractions | same pinned configuration | **9 %** |
| utterances landing 5.0–6.8 s after the previous `generationComplete`, 12 observations | **no `automaticActivityDetection` block at all** (server default) | **42 %** (5/12) |
| same risk band, 29 observations, later campaign | server default | **24 %** (7/29) |

**Caveat we would repeat to anyone quoting these numbers:** the rate oscillates strongly at fixed
configuration. Two consecutive sessions of one campaign gave **0 swallows out of 8**, and another
pair shortly afterwards gave **3 out of 10**, same config, same audio, same minutes. Treat these
as orders of magnitude, not point estimates.

Recipe that gets you there stochastically, with the config above:

1. Stream 16 kHz PCM continuously, 20 ms frames, zeros during pauses, never stopping.
2. Have the caller speak ~1.8 s utterances, each starting about **5.0–6.8 s after the previous
`generationComplete`**. That band is where we see the ignored utterances concentrate.
3. Roughly one utterance in three or four gets no reaction at all. When it does, **stay silent**
and watch the socket: that is the state this report is about.
4. Then speak again and watch the transcription of the *new* utterance close the *old* turn with
`out=None`.

Two negative results from the same work, so nobody re-runs them:

- `turnCoverage: TURN_INCLUDES_ALL_INPUT` changes neither the rate (3/10, 30 %, against 7/26,
27 %, measured in the same minutes) nor the window (8.01 / 8.03 / 8.01 s against 8.03 / 8.17 s).
- `startOfSpeechSensitivity=HIGH` (4/13, 31 %) and `prefixPaddingMs=150` (4/14, 29 %) do not
separate from the 42 % baseline at this sample size; inside the `prefixPaddingMs` arm alone the
rate goes from 0/5 in one round to 2/3 in another at identical parameters.

### The audio stream is continuous — this is measured, not asserted

This is the first objection raised on similar reports (e.g.
[dotnet-genai#269](https://github.com/googleapis/dotnet-genai/issues/269), closed after the
reporter's stream turned out to have gaps), so:

- Every probe here uses the same caller leg: one 20 ms PCM frame every 20 ms for the whole
session, digital zeros during pauses, never stopping — deliberately modelled on a telephone
line, which never goes silent on the wire.
- The standalone reproducer of our other report ([#37](https://github.com/google-gemini/gemini-live-api-examples/issues/37), script and output in https://gist.github.com/IngLP/124873f0574ada091ab4324e5ff12432) instruments that
sender and prints the numbers. Over a 63.3 s session: **3150 chunks sent, ~3166 expected, mean
inter-chunk interval 20.0 ms, worst interval 22.2 ms**, against a 20 ms nominal period. There is
no gap anywhere for the server to read as end of input.
- Independent cross-check on the same client, same audio path, same network: with
`automaticActivityDetection.disabled = true` and explicit `activityStart` / `activityEnd`
markers driven by a local VAD, the swallow rate in our lab drops from 29 % / 9 % to **1 % / 0 %**,
and **115 turns out of 115** were answered across three long sessions. Only the segmentation
authority changed. (We are not running that mode in production because it costs us ~2.3–2.4 s
of perceived latency against ~1.74 s in automatic mode — mentioning it only as evidence about
where the fault sits.)

### Related issues

The ignored-utterance half, reported in pieces:

- [cookbook#1197](https://github.com/google-gemini/cookbook/issues/1197) "Issue 9" — 567 audio chunks received, 0 `inputTranscription` events on an 11 s call
- [live-api-web-console#142](https://github.com/google-gemini/live-api-web-console/issues/142) — short utterances never committed by the VAD; extended to 3.1 in a 2026-04-25 comment
- [python-genai#2580](https://github.com/googleapis/python-genai/issues/2580) — `silence_duration_ms` ignored on 3.1
- [forum 146852](https://discuss.ai.google.dev/t/bug-gemini-3-1-flash-live-vad-does-not-process-audio-for-first-10-17-seconds-of-session/146852) — VAD deaf for the first 10–17 s of a session
- [forum 144747](https://discuss.ai.google.dev/t/gemini-3-1-flash-live-audio-input-via-websocket-never-triggers-a-response/144747) — audio in, only `sessionResumptionUpdate` out
- [python-genai#2333](https://github.com/googleapis/python-genai/issues/2333) (closed by the stale bot) — phantom `interrupted=True` wedging the session into a zero-audio state

We could not find the never-closing turn, or the next utterance being consumed to close it empty,
in any of these.

We are filing two other reports on the same model. They may share a root cause with this one, but
we have not demonstrated that: [#37](https://github.com/google-gemini/gemini-live-api-examples/issues/37), and a comment with a `voice_activity` repro on
[python-genai#1285](https://github.com/googleapis/python-genai/issues/1285). The missing
`voice_activity` signal is what makes this particular state undiagnosable from the client: there
is no server-side event that says whether the turn is still open.

### Questions

1. Is a turn opened on caller audio expected to remain open indefinitely when the server produces
no transcription and no generation for it?
2. Is `audioStreamEnd` supposed to have an effect in automatic-VAD mode? If it is a no-op there
by design, the documentation currently reads as though it flushes cached input audio.
3. Is there any client-observable signal that distinguishes "the server is still processing this
utterance" from "this utterance will never be answered"? Today we detect it only by absence,
with a timeout.

No repository file or test is identified. Start with the linked issue #37 reproducer and its gist, then investigate the documented automatic-VAD and audioStreamEnd behavior using the stated Python SDK configuration. Done means establishing whether the behavior is a service bug or documented design and answering the three reported questions with a reproducible diagnosis or corrective path.

Written by the indexing model from the issue text.