cloudflare / cloudflare/agents
voice ⨉ think/ai-chat
- Dominant language
- TypeScript
- Stars
- 5.6k
- Forks
- 711
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 53
Description
Parent roadmap: #1439
## Summary
Build a proof point that `@cloudflare/voice` works cleanly with Think / AI chat surfaces: voice input, turn detection, tool calling, approvals, interruption, and spoken responses should all compose without special-case app code.
This is not a Think stable blocker. Treat it as an integration/demo track that proves Think can support richer interaction modes after the core text/chat experience is solid.
## Why this matters
Voice is a strong pressure test for long-lived agents:
- turns are streaming and interruptible
- STT/VAD/turn detection can race with LLM/tool execution
- responses may be text, tool calls, or spoken audio
- tool calls and approvals need usable UX while the user is in a voice session
- persistent memory should remain consistent whether the user typed or spoke
If Think can support voice without bespoke wiring, that is a good sign that its chat/session/tool abstractions are healthy.
## Existing pieces
- `@cloudflare/voice` has `withVoice`, `withVoiceInput`, `useVoiceAgent`, and `useVoiceInput`.
- `docs/voice.md` documents continuous STT, TTS, interruption, conversation persistence, hooks, and React/client APIs.
- `design/voice.md` documents the experimental pipeline: one WebSocket, per-connection audio state, model-driven turn detection, interrupt handling, hibernation behavior, and transport/audio-input abstractions.
- `examples/voice-input` demonstrates STT-only dictation.
- `examples/elevenlabs-starter` demonstrates richer voice chat and media flows.
- Related bug: #1364 (`withVoice`: `onTurn` returning AI SDK `result.textStream` produces no TTS audio).
## What we need to prove
- A Think-like agent can accept voice input and route the resulting transcript through the same turn path as text chat.
- Tool calling works from a voice turn.
- Tool approval works during a voice turn with a clear UI state.
- Interruptions cancel active LLM/TTS work without corrupting Session/Think message state.
- VAD/STT end-of-turn behavior produces sensible chat turns and does not double-submit.
- Text and voice clients can observe consistent history for the same agent/session, or limitations are documented.
- Hibernation/reconnect behavior is acceptable for active and inactive calls.
## Design questions
- Should Think directly support `withVoice(Think)` / a `VoiceThink` pattern, or should voice remain a sibling package with examples?
- Does a voice transcript enter Think as a normal user message, a distinct multimodal/audio message, or both?
- How should pending approvals be surfaced in voice UX? Spoken prompt, visual approval card, both?
- If a tool result arrives after an interruption, should the turn continue, cancel, or ask the user?
- How should `chatRecovery` interact with active voice turns and TTS streaming?
- Can a voice turn call client-side tools, or should voice demos start with server tools only?
- What is the minimum provider story: Workers AI STT/TTS only, or also ElevenLabs/Telnyx adapters?
## Suggested demo shape
Create or extend an example that shows:
1. User speaks a request.
2. STT transcript appears in the chat history.
3. Think/AI chat executes a normal server tool.
4. A second tool requires approval; UI shows approve/reject while voice session remains active.
5. Approved result continues the turn.
6. Assistant response is spoken with TTS and persisted as a normal assistant message.
7. User interrupts playback; the active pipeline stops cleanly.
## Acceptance criteria
- Documented example proving voice + chat + tools + approval works end to end.
- #1364 or equivalent streaming-TTS issue is resolved or explicitly worked around.
- Clear docs on what is supported today and what remains experimental.
- No requirement that this block Think stable; this is a follow-up/flagship capability.
Contributor guide
Assessment
This issue has not been assessed yet.