[Feature] Multimodal content blocks in ChatMessage
- Dominant language
- Java
- Stars
- 452
- Forks
- 167
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 49
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
Tracking issue for the "Support invoking multimodal models" 0.4 Must item from #862. Design discussed and agreed in #1031.
Agreed direction:
* Replace `ChatMessage.content: String` with `blocks: List` in both Java and Python. No dual representation. Existing text factories continue to work by creating a single `TextBlock`, and `getText()` / `.text` provides the ordered text projection.
* `ContentBlock` includes `TextBlock` plus `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`. Media blocks share `mime_type`, exactly one of base64 `data` or external `url`, and optional metadata such as `name`, `size_bytes`, and `sha256`.
* Java and Python share the same serialized shape using a `type` discriminator (`text|image|audio|video|document`), covered by the cross-language snapshots and pemja bridge.
* Block ordering is preserved end-to-end. TOOL messages use the same block model, which also provides the path to fix MCP image tool results being stringified today.
* Each `ChatModelConnection` maps blocks to its provider-native representation. Unsupported block types fail explicitly rather than being silently dropped or converted.
* Event Log records media metadata only at every log level, including VERBOSE. Inline payloads are omitted and URL query parameters are stripped.
* `tool_calls` and `extra_args` stay unchanged here. Their cleanup remains tracked in #1056.
Subtasks:
**Phase 1: framework**
* [ ] Add the `ContentBlock` model and migrate `ChatMessage` in Java and Python
* [ ] Update factories, text projection, prompt/message merging, and repo call sites
* [ ] Update chat-event serialization, cross-language snapshots, and pemja bridge
* [ ] Add Event Log media redaction and URL sanitization
**Phase 2: providers, 0.4 target**
* [ ] OpenAI Chat Completions support in Java and Python, covering OpenAI, Azure OpenAI, and vLLM
* [ ] Ollama support in Java and Python
* [ ] Explicit unsupported-block errors for providers not yet migrated
**Follow-ups**
* [ ] OpenAI Responses API
* [ ] Anthropic, replacing the existing `anthropic_content_blocks` workaround
* [ ] Gemini, Bedrock, and Tongyi
* [ ] MCP image tool results as TOOL message blocks
* [ ] Multimodal YAML prompts
See #1031 for the full design discussion and rationale.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.