google-gemini / google-gemini/live-api-web-console

VAD configuration for short utterances - "Yes", "No", single names not triggering response

Open
#142 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.6k
Forks
731
PR merge metrics
No merged PRs in 30d

Description

### Description of the feature request:

## Summary

When using Gemini 2.5 Flash Native Audio via the Multimodal Live API in a **server-side telephony scenario** (Twilio → WebSocket → Gemini Live), short utterances such as
"Yes", "No", "John", or other single-word responses frequently fail to trigger a model response.

Longer utterances (>2 seconds) work reliably, but brief responses that are common in natural phone conversations do not.

## Environment

- **Model**: `gemini-2.5-flash-native-audio-preview-12-2025`
- **API**: Multimodal Live API (WebSocket)
- **Audio Input**: PCM 16-bit, 16kHz, mono from Twilio phone calls
- **Audio Chunk Size**: ~20ms per chunk (640 bytes)
- **Use Case**: Restaurant phone ordering system

## Current Configuration

```python
setup_message = {
"setup": {
"model": "models/gemini-2.5-flash-native-audio-preview-12-2025",
"generation_config": {
"response_modalities": ["AUDIO"],
"thinking_config": {
"thinking_budget": 0
},
"speech_config": {
"voice_config": {
"prebuilt_voice_config": {
"voice_name": "Kore"
}
}
}
},
"system_instruction": {
"parts": [{"text": "You are a friendly restaurant assistant..."}]
},
"tools": [...]
}
}
```

## Reproduction Steps

1. Customer calls Twilio phone number
2. Audio streams via WebSocket to our server
3. Server forwards PCM audio chunks to Gemini Live API via `realtimeInput.mediaChunks`
4. Customer says a short response like:
- "Yes"
- "No"
- "John" (their name)
- "Sure"
- "Okay"
5. **Expected**: Gemini responds with audio
6. **Actual**: No response is generated (most of the time)

## What Works

✅ Longer utterances (>2 seconds) trigger responses reliably
✅ Questions or full sentences work fine
✅ Azure Speech SDK STT recognizes these short utterances perfectly in parallel

## Questions / Feature Requests

Since the documentation for VAD and turn detection parameters is not publicly available, I'd like to request:

### 1. **VAD Configuration Parameters**
Are there any Voice Activity Detection parameters we can configure? For example:
```python
"speech_config": {
"vad_sensitivity": "low", # or "medium", "high"
"min_speech_duration_ms": 300,
"speech_end_timeout_ms": 800
}
```

### 2. **Turn Detection Configuration**
Can we adjust turn-taking/end-of-utterance detection? Similar to OpenAI's Realtime API:
```python
"turn_detection": {
"enabled": True,
"timeout_ms": 1500,
"silence_threshold_ms": 500
}
```

### 3. **Manual `turnComplete` Signal**
Should we send explicit `turnComplete` signals after detecting silence? What's the correct format?
```python
{
"clientContent": {
"turnComplete": True
}
}
```
or
```python
{
"realtimeInput": {
"turnComplete": True
}
}
```

### 4. **Minimum Audio Length Threshold**
Is there a minimum audio duration requirement? Should we pad short utterances with silence?

### 5. **Complete `generation_config` Schema**
Could you publish the complete configuration schema including all experimental parameters for `speech_config`, especially for server-side telephony use cases?

## Workaround Considered

We could use Azure STT for transcription and send text via `clientContent`, but this would lose the benefit of native audio processing:

```python
# Hybrid approach
user_text = azure_stt.recognize(audio)
await gemini_live.send_text(user_text) # Send as text instead of audio
```

However, this defeats the purpose of using Native Audio and adds latency.

## Impact

This issue significantly impacts conversational AI in telephony scenarios where:
- Short confirmations are common ("Yes", "No", "Okay")
- Customers provide brief information (names, numbers)
- Natural conversation flow requires quick back-and-forth

## Additional Context

- The same audio streams work perfectly with Azure Speech STT (no missed short utterances)
- This only occurs with audio input; text input via `clientContent` works for all lengths
- GitHub TypeScript example comments suggest VAD parameters exist but aren't documented

## Related Links

- [Multimodal Live API docs](https://ai.google.dev/api/multimodal-live) (VAD config returns 404)
- Similar pattern in OpenAI Realtime API: [Turn Detection](https://platform.openai.com/docs/guides/realtime/overview)

---

Thank you for considering this feature request! Happy to provide more details or test experimental parameters if needed.

### What problem are you trying to solve with this feature?

_No response_

### Any other information you'd like to share?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.