microsoft / microsoft/agent-framework
Python: Bug: Multimodal content is dropped during agent handoff in orchestrator
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
### Bug Description
In the current implementation of `clean_conversation_for_handoff`, the framework filters the conversation history by exclusively preserving `text` content types and discarding all other content.
While this effectively removes tool-control messages prior to routing, it inadvertently causes multi-modal content (such as `uri` or `data` attachments representing images, audio, or PDFs) to be silently dropped during agent handoffs. As a result, the receiving agent is provided with the text context but loses access to any associated multi-modal files from previous interactions.
### Impact
This behavior affects multi-modal workflows within `SequentialBuilder` and `HandoffAgentExecutor` pipelines. Specifically, it prevents Vision models from accessing images uploaded in prior steps, as the multi-modal context is stripped out during the handoff process.
### Proposed Solution
To resolve this, I propose updating the strict `text` check to a comprehensive allowlist. This approach will safely preserve necessary semantic content (such as `text`, `data`, `uri`, and `hosted_file`) while continuing to reliably filter out runtime and control payloads (such as `function_call`, `usage`, and `error`).
*I have prepared a Pull Request that addresses this issue, complete with updated test coverage in `test_handoff.py`. Thank you for your time and review!*
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.