agentscope-ai / agentscope-ai/agentscope

perf(realtime): improve the current realtime agent implementation

Open
#2,563 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31.5k
Forks
3.5k
Avg merge
1d 23h
Merged PRs (30d)
95

Description

## Background

Realtime agent is an increasingly important topic in LLM-based agent scenarios. Compared with traditional text-based communication, voice interaction offers a more natural, effortless, and efficient way for users to engage with AI systems. We believe LLM-powered agents will continue to evolve in this direction — a trend already reflected by major providers such as DashScope, Gemini, and OpenAI rolling out dedicated Realtime APIs, as well as tools like Codex and Claude Code beginning to incorporate voice interaction capabilities.

To this end, we aim to further explore how to build better realtime agents within the AgentScope framework, with a focus on the following dimensions:

- **Latency & Performance**: traditional voice interaction metrics such as TTFB (Time to First Byte/Audio), end-to-end latency, VAD response time, and barge-in handling
- **Cascaded Pipeline (ASR → LLM → TTS)**: a classical and flexible architecture that chains independent speech recognition, language model reasoning, and speech synthesis components, where each stage can be independently swapped or upgraded, and streaming across stages helps minimize perceived latency
- **Context Management**: strategies for maintaining, compressing, and recovering conversation context across long or interrupted sessions
- **Interaction Naturalness**: fluency of turn-taking, prosody quality, and overall conversational experience

We plan to deliver an improved realtime agent implementation in the **v2.1.0** release.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.