OpenBMB / OpenBMB/MiniCPM-o-Demo

Structured tool calling for Audio Full-Duplex Realtime API — qwen-audio-agent integration

Open
#62 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
386
Forks
81
Avg merge
1h 59m
Merged PRs (30d)
3

Description

Thank you for open-sourcing MiniCPM-o 4.5 and its Realtime inference service!

We have integrated MiniCPM-o 4.5 into qwen-audio-agent as a realtime voice frontend, so users can talk naturally while the framework coordinates backend agents.

The integration is available in qwen-audio-agent PR #344.

From the currently documented Audio Full-Duplex protocol, we found session lifecycle and text/audio streaming events, but no structured events for tool-call requests or tool-result submission.

Could you please clarify:

  1. Does the current full-duplex Realtime API expose any tool/function-calling interface that is not yet documented?
  2. If not, is structured tool calling planned at the protocol level in a future release, for example function-call arguments plus tool-result submission?

We saw the model capability discussion in #21. This issue focuses specifically on the Realtime wire protocol needed by host applications.

This capability would be especially valuable for realtime voice agent runtimes: MiniCPM-o could decide when to call a tool, while the host executes it and returns the result without interrupting the voice interaction.

We would be happy to test and provide integration feedback when the capability becomes available.

Thanks again for the excellent open-source work!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the documented Audio Full-Duplex protocol and qwen-audio-agent PR #344; compare the existing session lifecycle and text/audio streaming events with the requested tool-call and tool-result flow. Review issue #21 for related capability context. Done means a maintainer-confirmed protocol answer or an agreed implementation scope, which this issue does not yet define.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.