googleapis / googleapis/python-genai
Live API (gemini-3.1-flash-live-preview): VAD silently drops short user utterances right after a model turn (telephony, audio provably delivered)
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 40
Description
**Note: we use the Live API over the raw BidiGenerateContent WebSocket (not this SDK), but filing here as Live API model-behaviour reports appear to be triaged in this repo (cf. #1894). Happy to move this wherever it belongs.**
### Summary
`models/gemini-3.1-flash-live-preview` (Live API, audio-to-audio, telephony use case) consistently **ignores short user utterances (~1–2s) spoken shortly after the model finishes its own turn**. No `inputTranscription` is produced for them, no turn is opened, no error is returned — the session just stays silent until the user produces a *longer* utterance. Callers experience this as "the agent suddenly went away" and hang up.
### Evidence (production call, 2026-08-24, ~08:01–08:02 UTC)
Timeline reconstructed from our bridge logs + the dual-channel call recording:
| t (rel) | Channel | Event |
|---|---|---|
| 60.8s | model | Model finishes turn (asks "shall I continue?"), `turnComplete` received |
| 61–62s | caller | **"Ja, dat hoort."** — clear speech (verified, see below) → **no VAD trigger, no transcription, no response** |
| 68–69s | caller | **"Ben je er nog?"** — clear speech → **ignored again** |
| 74–79s | caller | Longer complaint sentence (~5s) → turn finally triggers, model responds normally |
Three independent verifications:
1. **Audio was delivered continuously.** Our bridge streams 20ms PCM16/16k chunks (converted from 8kHz μ-law telephony audio) via `realtimeInput.audio`; send counters show an uninterrupted ~50 chunks/sec throughout the window, no WS errors, no reconnects, no `goAway`.
2. **The speech is objectively clear.** The same inbound audio (from the call recording, same leg) fed to `gemini-3.7-flash` `generateContent` transcribes perfectly, including timestamps: `[0:01-0:03] (clear speech): "Ja, dat hoort."` / `[0:09-0:10] (clear speech): "Ben je er nog?"`. So Gemini's own STT understands this audio — only the Live VAD gate in front of it drops it.
3. **VAD was configured at maximum sensitivity** at the time of the call:
```json
"realtimeInputConfig": {
"automaticActivityDetection": {
"startOfSpeechSensitivity": "START_SENSITIVITY_HIGH",
"endOfSpeechSensitivity": "END_SENSITIVITY_HIGH",
"prefixPaddingMs": 300,
"silenceDurationMs": 800
}
}
```
The same pattern occurred on earlier calls (e.g. 2026-08-22: a short "Ja." confirming a question was ignored for 25s until the caller spoke a longer sentence) — also with default VAD settings, so it is not caused by this particular tuning.
### Setup
- Model: `models/gemini-3.1-flash-live-preview`, raw WebSocket `v1beta.GenerativeService.BidiGenerateContent`
- Audio in: PCM16 @16kHz (upsampled from 8kHz μ-law telephony), 20ms chunks, sent continuously (no client-side gating)
- `responseModalities: ["AUDIO"]`, `speechConfig.languageCode: "nl-NL"`, input/output transcription enabled (`languageCodes: ["nl-NL"]`), `sessionResumption` + `contextWindowCompression.slidingWindow` enabled, ~15 function declarations + googleSearch
- System instruction ~9k chars
### Expected
A clear 1–2s utterance after the model's turn should open a user turn (or at minimum produce an `inputTranscription`), especially with `START_SENSITIVITY_HIGH`.
### Actual
Short utterances right after a model turn are silently dropped; longer utterances work. This makes natural short confirmations ("ja", "nee", "klopt") — the most common turn type in phone conversations — unreliable.
We can share the dual-channel recording and full logs privately on request.
Contributor guide
Assessment
This issue has not been assessed yet.