microsoft / microsoft/agent-framework
Single Agent: RealtimeAgent and client implementations for integration with realtime APIs
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
Following from the [experimental service in Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/realtime?pivots=programming-language-python) and the [recent GA of gpt-realtime and the Realtime API](https://openai.com/index/introducing-gpt-realtime/), we should have flexible integrations with this and other realtime models/services inc. [Gemini Live API](https://ai.google.dev/gemini-api/docs/live) and [Azure Voice Live](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live)
To keep this close to the current interfaces, this could be a new class of agent (`RealtimeAgent`) with associated realtime clients for the service of choice (i.e. `OpenAIRealtimeClient`, `AzureRealtimeClient`). Using OpenAI's [Agents SDK realtime implementation](https://openai.github.io/openai-agents-js/openai/agents-realtime/classes/realtimeagent/) as inspiration, we would need a concept of a `Session` as well, and this would need to be updated upon agent handoffs when used in an orchestration pattern.
Contributor guide
Research direction
No files, tests, or implementation entry points are named. Start by reviewing the current agent interfaces and the linked realtime implementations and API documentation, then clarify the scope for RealtimeAgent, service-specific clients, Session handling, and handoffs before defining completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- ai, api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100