agentscope-ai / agentscope-ai/agentscope
perf(realtime): improve the current realtime agent implementation
- Dominant language
- Python
- Stars
- 31.5k
- Forks
- 3.5k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 95
Description
## Background
Realtime agent is an increasingly important topic in LLM-based agent scenarios. Compared with traditional text-based communication, voice interaction offers a more natural, effortless, and efficient way for users to engage with AI systems. We believe LLM-powered agents will continue to evolve in this direction — a trend already reflected by major providers such as DashScope, Gemini, and OpenAI rolling out dedicated Realtime APIs, as well as tools like Codex and Claude Code beginning to incorporate voice interaction capabilities.
To this end, we aim to further explore how to build better realtime agents within the AgentScope framework, with a focus on the following dimensions:
- **Latency & Performance**: traditional voice interaction metrics such as TTFB (Time to First Byte/Audio), end-to-end latency, VAD response time, and barge-in handling
- **Cascaded Pipeline (ASR → LLM → TTS)**: a classical and flexible architecture that chains independent speech recognition, language model reasoning, and speech synthesis components, where each stage can be independently swapped or upgraded, and streaming across stages helps minimize perceived latency
- **Context Management**: strategies for maintaining, compressing, and recovering conversation context across long or interrupted sessions
- **Interaction Naturalness**: fluency of turn-taking, prosody quality, and overall conversational experience
We plan to deliver an improved realtime agent implementation in the **v2.1.0** release.
Contributor guide
Assessment
This issue has not been assessed yet.