googleapis / googleapis/python-genai

Live API (gemini-3.1-flash-live-preview): VAD silently drops short user utterances right after a model turn (telephony, audio provably delivered)

Open
#2,903 2 comments 0 reactions 1 assignee Claimed by @kkorpal View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

**Note: we use the Live API over the raw BidiGenerateContent WebSocket (not this SDK), but filing here as Live API model-behaviour reports appear to be triaged in this repo (cf. #1894). Happy to move this wherever it belongs.**

### Summary

`models/gemini-3.1-flash-live-preview` (Live API, audio-to-audio, telephony use case) consistently **ignores short user utterances (~1–2s) spoken shortly after the model finishes its own turn**. No `inputTranscription` is produced for them, no turn is opened, no error is returned — the session just stays silent until the user produces a *longer* utterance. Callers experience this as "the agent suddenly went away" and hang up.

### Evidence (production call, 2026-08-24, ~08:01–08:02 UTC)

Timeline reconstructed from our bridge logs + the dual-channel call recording:

| t (rel) | Channel | Event |
|---|---|---|
| 60.8s | model | Model finishes turn (asks "shall I continue?"), `turnComplete` received |
| 61–62s | caller | **"Ja, dat hoort."** — clear speech (verified, see below) → **no VAD trigger, no transcription, no response** |
| 68–69s | caller | **"Ben je er nog?"** — clear speech → **ignored again** |
| 74–79s | caller | Longer complaint sentence (~5s) → turn finally triggers, model responds normally |

Three independent verifications:

1. **Audio was delivered continuously.** Our bridge streams 20ms PCM16/16k chunks (converted from 8kHz μ-law telephony audio) via `realtimeInput.audio`; send counters show an uninterrupted ~50 chunks/sec throughout the window, no WS errors, no reconnects, no `goAway`.
2. **The speech is objectively clear.** The same inbound audio (from the call recording, same leg) fed to `gemini-3.7-flash` `generateContent` transcribes perfectly, including timestamps: `[0:01-0:03] (clear speech): "Ja, dat hoort."` / `[0:09-0:10] (clear speech): "Ben je er nog?"`. So Gemini's own STT understands this audio — only the Live VAD gate in front of it drops it.
3. **VAD was configured at maximum sensitivity** at the time of the call:

```json
"realtimeInputConfig": {
"automaticActivityDetection": {
"startOfSpeechSensitivity": "START_SENSITIVITY_HIGH",
"endOfSpeechSensitivity": "END_SENSITIVITY_HIGH",
"prefixPaddingMs": 300,
"silenceDurationMs": 800
}
}
```

The same pattern occurred on earlier calls (e.g. 2026-08-22: a short "Ja." confirming a question was ignored for 25s until the caller spoke a longer sentence) — also with default VAD settings, so it is not caused by this particular tuning.

### Setup

- Model: `models/gemini-3.1-flash-live-preview`, raw WebSocket `v1beta.GenerativeService.BidiGenerateContent`
- Audio in: PCM16 @16kHz (upsampled from 8kHz μ-law telephony), 20ms chunks, sent continuously (no client-side gating)
- `responseModalities: ["AUDIO"]`, `speechConfig.languageCode: "nl-NL"`, input/output transcription enabled (`languageCodes: ["nl-NL"]`), `sessionResumption` + `contextWindowCompression.slidingWindow` enabled, ~15 function declarations + googleSearch
- System instruction ~9k chars

### Expected

A clear 1–2s utterance after the model's turn should open a user turn (or at minimum produce an `inputTranscription`), especially with `START_SENSITIVITY_HIGH`.

### Actual

Short utterances right after a model turn are silently dropped; longer utterances work. This makes natural short confirmations ("ja", "nee", "klopt") — the most common turn type in phone conversations — unreliable.

We can share the dual-channel recording and full logs privately on request.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.