googleapis / googleapis/python-genai

Add `speech_started` and `speech_stopped` events for automatic VAD in Gemini Live

Aperta
#1,285 17 commenti 6 reazioni 1 assegnatario Rivendicata da @Venkaiahbabuneelam Vedi su GitHub
api: gemini-api priority: p3 type: feature request
Lingua principale
Python
Stelle
4k
Fork
1k
Merge medio
2g 12h
PR unite (30g)
41

Descrizione

**The problem**
When using Gemini Live with **automatic voice activity detection (VAD)**, there is no way for the client to know when the model detects the *start* or *end* of user speech.
OpenAI’s Realtime API provides two very useful events for this:

* `input_audio_buffer.speech_started` → emitted as soon as the model’s VAD detects that the user has begun speaking.
* `input_audio_buffer.speech_stopped` → emitted as soon as the model’s VAD detects that the user has stopped speaking.

These signals are important for building real-time experiences (e.g., showing microphone indicators, animating avatars, syncing captions, or debugging latency).

Gemini Live does not expose such events today. The only related flag is `interrupted`, but that serves a different purpose (signaling that the *model’s response* was cut off by user speech, not speech boundary detection).

---

**The solution**
Please add support for Gemini Live responses to include explicit events or fields for:

* `speech_started` — when automatic VAD detects user speech onset.
* `speech_stopped` — when automatic VAD detects user speech offset.

This would mirror OpenAI Realtime’s design and make it much easier to integrate Gemini Live into responsive UIs without requiring developers to implement their own manual VAD logic.

---

**Alternatives:**

* Using manual VAD mode (`activityStart` / `activityEnd`) and sending these signals ourselves. However, this defeats the purpose of using automatic VAD, since it requires implementing and maintaining our own VAD pipeline.
* Inferring speech start/stop indirectly (e.g., based on when Gemini responds), which is unreliable and adds latency.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.