microsoft / microsoft/agent-framework

Single Agent: RealtimeAgent and client implementations for integration with realtime APIs

Open
#728 9 comments 14 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

Following from the [experimental service in Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/realtime?pivots=programming-language-python) and the [recent GA of gpt-realtime and the Realtime API](https://openai.com/index/introducing-gpt-realtime/), we should have flexible integrations with this and other realtime models/services inc. [Gemini Live API](https://ai.google.dev/gemini-api/docs/live) and [Azure Voice Live](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live)

To keep this close to the current interfaces, this could be a new class of agent (`RealtimeAgent`) with associated realtime clients for the service of choice (i.e. `OpenAIRealtimeClient`, `AzureRealtimeClient`). Using OpenAI's [Agents SDK realtime implementation](https://openai.github.io/openai-agents-js/openai/agents-realtime/classes/realtimeagent/) as inspiration, we would need a concept of a `Session` as well, and this would need to be updated upon agent handoffs when used in an orchestration pattern.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or implementation entry points are named. Start by reviewing the current agent interfaces and the linked realtime implementations and API documentation, then clarify the scope for RealtimeAgent, service-specific clients, Session handling, and handoffs before defining completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
ai, api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.