agentscope-ai / agentscope-ai/agentscope

feat(realtime): add cascaded realtime model (asr-llm-tts)

未關閉
#2,565 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
31.6k
分支
3.5k
平均合併
1 天 16 小時
30 天內合併 PR
103

描述

**Background**
The current `RealtimeModel` relies on native speech-to-speech APIs, which have limited provider coverage and controllability. A cascaded pipeline (ASR → LLM → TTS) is a natural alternative — more flexible, provider-agnostic, and easier to debug.

The proposed approach is to implement a `CascadedRealtimeModel` that shares the same interface as the existing `RealtimeModel`, so that `RealtimeAgent` can switch between speech-to-speech and cascaded mode purely by swapping the `model` field. This design assumption needs to be validated against the current `RealtimeModel` interface before implementation.

> Depends on #2564 (ASR module abstraction) to be merged first.

**Changes**
- Implement `CascadedRealtimeModel` compatible with the existing `RealtimeModel` interface
- `RealtimeAgent` switches between speech-to-speech and cascaded mode via the `model` field only — no agent-level changes required
- Support streaming across all three stages to minimize end-to-end latency

貢獻指南

開啟貢獻指南

研究方向

Start with the `RealtimeModel` and `RealtimeAgent` definitions to confirm their current contract and how the `model` field is consumed. Validate the existing streaming path in the realtime flow and check issue #2564’s ASR abstraction context before drafting the new pipeline. Implement `CascadedRealtimeModel` as a drop-in model with the same interface so mode selection is only via `model`, and verify streaming behavior across ASR→LLM→TTS stays intact without RealtimeAgent code changes.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
machine-learning
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
44/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。