android / android/appfunctions
[FR]: Gemini Live API integration for voice-based interaction
- Dominant language
- Kotlin
- Stars
- 191
- Forks
- 36
- Avg merge
- 13d 16h
- Merged PRs (30d)
- 2
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the problem
AppFunctions Studio currently only supports text-based interaction with the agent. For interactive showcases, being able to speak to the agent and hear responses would be dramatically more compelling and natural — especially when demonstrating how AppFunctions enable hands-free, voice-driven workflows on Android.
### Describe the solution
#### Motivation
AppFunctions are fundamentally about enabling agent-driven automation without opening app UIs. The most natural interface for this is voice — a user says "create a note about tomorrow's meeting" and the agent executes it via AppFunctions. But the current testing agent forces users to type, which undermines the demo narrative.
#### Key use cases:
- Demos: Showing executives how AppFunctions work in a hands-free, conversational way
- Developer testing: Rapidly testing AppFunction flows via voice without typing complex queries
- Accessibility: Voice-first interaction for users who can't easily type on mobile
#### Technical Context
The agent currently uses the Interactions API with text input exclusively. Google already offers the [Gemini Live API](https://ai.google.dev/gemini-api/docs/live-api) which supports:
- Real-time audio input/output (bidirectional streaming)
- Tool use / function calling — critical for AppFunctions execution
- Low-latency audio-to-audio via gemini-3.1-flash-live-preview
- Session management for multi-turn conversations
The Live API already supports tools, meaning the existing AppFunction → Gemini tool schema conversion could be reused. The agent would need:
1. Audio capture (microphone) → streaming to Live API
2. Live API processes speech, calls AppFunctions as tools (same flow as today)
3. Audio response streamed back → speaker output
#### Suggested Implementation
- Add a "Voice mode" toggle or a push-to-talk button in the chat UI
- Show a transcript in the chat alongside audio (for clarity and debugging)
### Additional context
_No response_
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start by tracing the current text-only Interactions API flow in the chat UI and the existing AppFunction-to-Gemini tool schema conversion mentioned in the issue. Review the Gemini Live API documentation and Android audio requirements before deciding how streaming, tool calls, transcripts, and voice-mode controls fit together. Done means voice input and output work end to end while AppFunctions remain callable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, kotlin
- Domain
- ai, audio-video-rtc, mobile
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100