api7 / api7/aisix

P0-8: Multimodal content blocks (vision, image input) on chat completions

Open
#50 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cross-repo gap-with-litellm P0
Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Problem

Bridge trait's chat path takes `content: String`. The OpenAI/Anthropic content-block format (mixed text + image_url + base64 image) cannot be expressed. Every vision-question-answering scenario fails at the type system.

LiteLLM ships native vision through Anthropic / OpenAI / Gemini / Bedrock multimodal models.

Scope

DP (moonming/ai-gateway)
  • Change Message.content from String to Vec<ContentBlock> where ContentBlock = Text(String) | ImageUrl { url, detail } | ImageBase64 { media_type, data }. Backward-compatible serde: a plain string deserializes to vec![ContentBlock::Text(s)].
  • Update OpenAI bridge (crates/aisix-provider-openai/src/wire.rs) to emit the OpenAI content-block JSON shape.
  • Update Anthropic bridge (crates/aisix-provider-anthropic/src/wire.rs) to emit {"type":"image","source":{"type":"base64",...}} blocks.
  • Gemini: Gemini's OpenAI-compat endpoint accepts the same shape (no change needed). Native generateContent (P1-7) will need its own image part shape.
  • Bedrock (after P0-2): map to Bedrock Converse content: [{image: {format, source: {bytes}}}].
CP (api7/api7ee-3-control-plane)
  • N/A. Data shape is opaque to CP.
Dashboard UI (api7/AISIX-Cloud)
  • Playground page: file-upload affordance for images → base64 in the request payload. Optional in v1 (text playground works).
e2e
  • Test: send a chat completion with an image_url block to OpenAI mock + assert mock received the block correctly.
  • Test: same for Anthropic mock.

Estimate

DP 5d (touches every provider crate); CP 0d; UI 1d; e2e 1d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Bridge trait and Message definition, then read crates/aisix-provider-openai/src/wire.rs and crates/aisix-provider-anthropic/src/wire.rs. Verify the OpenAI mock and Anthropic mock end-to-end cases first; the work is done when text, image_url, and base64 image blocks serialize correctly for both providers and the existing text path remains compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.