android / android/appfunctions

[FR]: Gemini Live API integration for voice-based interaction

Open
#5 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Kotlin
Stars
191
Forks
36
Avg merge
13d 16h
Merged PRs (30d)
2

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the problem

AppFunctions Studio currently only supports text-based interaction with the agent. For interactive showcases, being able to speak to the agent and hear responses would be dramatically more compelling and natural — especially when demonstrating how AppFunctions enable hands-free, voice-driven workflows on Android.

### Describe the solution

#### Motivation
AppFunctions are fundamentally about enabling agent-driven automation without opening app UIs. The most natural interface for this is voice — a user says "create a note about tomorrow's meeting" and the agent executes it via AppFunctions. But the current testing agent forces users to type, which undermines the demo narrative.

#### Key use cases:
- Demos: Showing executives how AppFunctions work in a hands-free, conversational way
- Developer testing: Rapidly testing AppFunction flows via voice without typing complex queries
- Accessibility: Voice-first interaction for users who can't easily type on mobile

#### Technical Context
The agent currently uses the Interactions API with text input exclusively. Google already offers the [Gemini Live API](https://ai.google.dev/gemini-api/docs/live-api) which supports:

- Real-time audio input/output (bidirectional streaming)
- Tool use / function calling — critical for AppFunctions execution
- Low-latency audio-to-audio via gemini-3.1-flash-live-preview
- Session management for multi-turn conversations

The Live API already supports tools, meaning the existing AppFunction → Gemini tool schema conversion could be reused. The agent would need:

1. Audio capture (microphone) → streaming to Live API
2. Live API processes speech, calls AppFunctions as tools (same flow as today)
3. Audio response streamed back → speaker output

#### Suggested Implementation

- Add a "Voice mode" toggle or a push-to-talk button in the chat UI
- Show a transcript in the chat alongside audio (for clarity and debugging)

### Additional context

_No response_

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start by tracing the current text-only Interactions API flow in the chat UI and the existing AppFunction-to-Gemini tool schema conversion mentioned in the issue. Review the Gemini Live API documentation and Android audio requirements before deciding how streaming, tool calls, transcripts, and voice-mode controls fit together. Done means voice input and output work end to end while AppFunctions remain callable.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, kotlin
Domain
ai, audio-video-rtc, mobile
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.