cloudflare / cloudflare/agents

Voice SDK: add native full-duplex provider and AI SDK tool lifecycle

Open
#2,158 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
5.6k
Forks
711
Avg merge
1d 20h
Merged PRs (30d)
53

Description

**Describe the gap**

@cloudflare/voice 0.3.5 exposes a Transcriber session for continuous STT and withVoice owns the cascaded onTurn plus TTS pipeline. A full-duplex speech provider such as Grok Voice, Gemini Live, or OpenAI Realtime emits user transcripts, assistant audio, interruptions, and function calls on one provider session.

There is no native full-duplex provider contract and no place to attach an AI SDK ToolSet to that session. Consumers must replace the transcriber role with a custom session adapter and implement function-call validation, execution ordering, result submission, and close settlement themselves.

**To reproduce**

1. Extend withVoice(Agent).
2. Connect a full-duplex realtime provider through createTranscriber.
3. Register the same AI SDK tools that another Agent surface uses.
4. Let the provider emit a function call while an earlier function call is still executing.
5. The Voice interfaces have no tool registry, execution lifecycle, serialization or drain contract, or function-result submission hook.

**Expected behavior**

Please add a native full-duplex provider session surface that:

- owns bidirectional audio, transcripts, playback interruption, and provider session close;
- accepts an AI SDK ToolSet so schemas, validation, and handlers remain canonical;
- serializes function calls or defines explicit drain semantics when provider events overlap;
- returns function results through the provider session before the next response;
- settles pending final transcripts and tool results before close; and
- permits first-party provider adapters without consumers replacing Voice lifecycle behavior.

This would let applications keep withVoice as the call owner and keep provider code as a narrow adapter instead of building a parallel voice and tool lifecycle.

**Screenshots**

Not applicable.

**Version**

- @cloudflare/voice: 0.3.5
- agents: 0.20.1
- ai: 7.0.58

**Additional context**

The existing Transcriber and TranscriberSession types are a good continuous-STT contract. The missing surface is the provider-native speech-to-speech and function-tool equivalent.

Contributor guide

Open the contributing guide

Research direction

Read the existing Transcriber and TranscriberSession types, then inspect the withVoice and createTranscriber entry points. Define the missing full-duplex session boundary and ToolSet lifecycle, including overlapping calls, result submission, interruptions, and close settlement; done means first-party adapters can use it without replacing Voice lifecycle behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, audio-video-rtc, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.