P0-8: Multimodal content blocks (vision, image input) on chat completions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 157
- Forks
- 32
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 145
Description
Problem
Bridge trait's chat path takes `content: String`. The OpenAI/Anthropic content-block format (mixed text + image_url + base64 image) cannot be expressed. Every vision-question-answering scenario fails at the type system.
LiteLLM ships native vision through Anthropic / OpenAI / Gemini / Bedrock multimodal models.
Scope
DP (moonming/ai-gateway)
- Change
Message.contentfromStringtoVec<ContentBlock>whereContentBlock = Text(String) | ImageUrl { url, detail } | ImageBase64 { media_type, data }. Backward-compatible serde: a plain string deserializes tovec![ContentBlock::Text(s)]. - Update OpenAI bridge (
crates/aisix-provider-openai/src/wire.rs) to emit the OpenAI content-block JSON shape. - Update Anthropic bridge (
crates/aisix-provider-anthropic/src/wire.rs) to emit{"type":"image","source":{"type":"base64",...}}blocks. - Gemini: Gemini's OpenAI-compat endpoint accepts the same shape (no change needed). Native generateContent (P1-7) will need its own image part shape.
- Bedrock (after P0-2): map to Bedrock Converse
content: [{image: {format, source: {bytes}}}].
CP (api7/api7ee-3-control-plane)
- N/A. Data shape is opaque to CP.
Dashboard UI (api7/AISIX-Cloud)
- Playground page: file-upload affordance for images → base64 in the request payload. Optional in v1 (text playground works).
e2e
- Test: send a chat completion with an image_url block to OpenAI mock + assert mock received the block correctly.
- Test: same for Anthropic mock.
Estimate
DP 5d (touches every provider crate); CP 0d; UI 1d; e2e 1d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Bridge trait and Message definition, then read crates/aisix-provider-openai/src/wire.rs and crates/aisix-provider-anthropic/src/wire.rs. Verify the OpenAI mock and Anthropic mock end-to-end cases first; the work is done when text, image_url, and base64 image blocks serialize correctly for both providers and the existing text path remains compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100