apache / apache/flink-agents

[Feature] Multimodal content blocks in ChatMessage

Open
#1,059 1 comment 0 reactions 1 assignee Claimed by @Zhuoxi2000 View on GitHub
feature priority/major
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.

### Description

Tracking issue for the "Support invoking multimodal models" 0.4 Must item from #862. Design discussed and agreed in #1031.

Agreed direction:

* Replace `ChatMessage.content: String` with `blocks: List` in both Java and Python. No dual representation. Existing text factories continue to work by creating a single `TextBlock`, and `getText()` / `.text` provides the ordered text projection.
* `ContentBlock` includes `TextBlock` plus `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`. Media blocks share `mime_type`, exactly one of base64 `data` or external `url`, and optional metadata such as `name`, `size_bytes`, and `sha256`.
* Java and Python share the same serialized shape using a `type` discriminator (`text|image|audio|video|document`), covered by the cross-language snapshots and pemja bridge.
* Block ordering is preserved end-to-end. TOOL messages use the same block model, which also provides the path to fix MCP image tool results being stringified today.
* Each `ChatModelConnection` maps blocks to its provider-native representation. Unsupported block types fail explicitly rather than being silently dropped or converted.
* Event Log records media metadata only at every log level, including VERBOSE. Inline payloads are omitted and URL query parameters are stripped.
* `tool_calls` and `extra_args` stay unchanged here. Their cleanup remains tracked in #1056.

Subtasks:

**Phase 1: framework**

* [ ] Add the `ContentBlock` model and migrate `ChatMessage` in Java and Python
* [ ] Update factories, text projection, prompt/message merging, and repo call sites
* [ ] Update chat-event serialization, cross-language snapshots, and pemja bridge
* [ ] Add Event Log media redaction and URL sanitization

**Phase 2: providers, 0.4 target**

* [ ] OpenAI Chat Completions support in Java and Python, covering OpenAI, Azure OpenAI, and vLLM
* [ ] Ollama support in Java and Python
* [ ] Explicit unsupported-block errors for providers not yet migrated

**Follow-ups**

* [ ] OpenAI Responses API
* [ ] Anthropic, replacing the existing `anthropic_content_blocks` workaround
* [ ] Gemini, Bedrock, and Tongyi
* [ ] MCP image tool results as TOOL message blocks
* [ ] Multimodal YAML prompts

See #1031 for the full design discussion and rationale.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.