microsoft / microsoft/agent-framework

Python: Bug: Multimodal content is dropped during agent handoff in orchestrator

Open
#7,822 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

orchestration python reproduced
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

### Bug Description
In the current implementation of `clean_conversation_for_handoff`, the framework filters the conversation history by exclusively preserving `text` content types and discarding all other content.

While this effectively removes tool-control messages prior to routing, it inadvertently causes multi-modal content (such as `uri` or `data` attachments representing images, audio, or PDFs) to be silently dropped during agent handoffs. As a result, the receiving agent is provided with the text context but loses access to any associated multi-modal files from previous interactions.

### Impact
This behavior affects multi-modal workflows within `SequentialBuilder` and `HandoffAgentExecutor` pipelines. Specifically, it prevents Vision models from accessing images uploaded in prior steps, as the multi-modal context is stripped out during the handoff process.

### Proposed Solution
To resolve this, I propose updating the strict `text` check to a comprehensive allowlist. This approach will safely preserve necessary semantic content (such as `text`, `data`, `uri`, and `hosted_file`) while continuing to reliably filter out runtime and control payloads (such as `function_call`, `usage`, and `error`).

*I have prepared a Pull Request that addresses this issue, complete with updated test coverage in `test_handoff.py`. Thank you for your time and review!*

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.