GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

[Bug]: Overlapping Audio Causes Misinterpreted User Input in WebSocket Demo App

Open
#1,817 0 comments 0 reactions 1 assignee Claimed by @ZackAkil View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

### File Name

gemini/multimodal-live-api/websocket-demo-app

### What happened?

When using the WebSocket Demo App in the Gemini Multimodal Live API repository, there is an issue where audio inputs overlap during user speech. This overlapping causes the app to mistakenly treat parts of the audio response as if they were user responses, thereby disrupting the natural conversation flow.

**Steps to Reproduce:**
1. Launch the WebSocket Demo App from the [generative-ai repository](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/gemini/multimodal-live-api/websocket-demo-app).
2. Start a conversation by speaking into the microphone.
3. During the conversation, observe that when the user is speaking, overlapping audio (from system responses) is captured concurrently.
4. Notice that audio segments received during the overlap are interpreted as user input, which causes the conversation to continue erroneously.

Please let me know if further details are needed to help diagnose this issue.

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.