googleapis / googleapis/python-genai
Add `speech_started` and `speech_stopped` events for automatic VAD in Gemini Live
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 40
Description
**The problem**
When using Gemini Live with **automatic voice activity detection (VAD)**, there is no way for the client to know when the model detects the *start* or *end* of user speech.
OpenAI’s Realtime API provides two very useful events for this:
* `input_audio_buffer.speech_started` → emitted as soon as the model’s VAD detects that the user has begun speaking.
* `input_audio_buffer.speech_stopped` → emitted as soon as the model’s VAD detects that the user has stopped speaking.
These signals are important for building real-time experiences (e.g., showing microphone indicators, animating avatars, syncing captions, or debugging latency).
Gemini Live does not expose such events today. The only related flag is `interrupted`, but that serves a different purpose (signaling that the *model’s response* was cut off by user speech, not speech boundary detection).
---
**The solution**
Please add support for Gemini Live responses to include explicit events or fields for:
* `speech_started` — when automatic VAD detects user speech onset.
* `speech_stopped` — when automatic VAD detects user speech offset.
This would mirror OpenAI Realtime’s design and make it much easier to integrate Gemini Live into responsive UIs without requiring developers to implement their own manual VAD logic.
---
**Alternatives:**
* Using manual VAD mode (`activityStart` / `activityEnd`) and sending these signals ourselves. However, this defeats the purpose of using automatic VAD, since it requires implementing and maintaining our own VAD pipeline.
* Inferring speech start/stop indirectly (e.g., based on when Gemini responds), which is unreliable and adds latency.
Contributor guide
Assessment
This issue has not been assessed yet.