[Feature]: model/openaimodel: support image/file input (genai InlineData/FileData parts)
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 1k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 88
Description
## Problem
The OpenAI model adapter (`model/openaimodel`) only supports **text** content parts. When a user passes an image or file (e.g. `genai.Part.InlineData` / `genai.Part.FileData`) to an agent using an OpenAI model, `convertContents` falls through to the `default` branch and returns:
```
openai: unsupported content part *genai.Part
```
Relevant code: `model/openaimodel/request.go:116` — the part switch in `convertContents` handles only `Text`, `FunctionCall`, and `FunctionResponse`. The behavior is even pinned by a test asserting failure: `TestBuildOpenAIParams_UnsupportedPart` (`model/openaimodel/request_test.go:123`).
## Expected behavior
Support OpenAI-format image/file input in the Responses API:
- `genai.Part.InlineData` (`*genai.Blob` with `Data` + `MimeType`) → `responses.ResponseInputImageParam` (`input_image`, base64 `data:...` or `file_id`) — covers images, and audio/video via `input_audio`/`input_video` where applicable.
- `genai.Part.FileData` (remote `URI` + `MimeType`) → `responses.ResponseInputFileParam` (`input_file`), e.g. PDFs.
- A message containing both text and image parts should be emitted as a single message with a content list mixing `input_text` and `input_image` items (the existing `newMessage` only builds `ResponseInputTextParam`).
## Repro sketch
```go
req := &model.LLMRequest{
Contents: []*genai.Content{{
Role: string(genai.RoleUser),
Parts: []*genai.Part{
{Text: "What is in this image?"},
{InlineData: &genai.Blob{MimeType: "image/png", Data: pngBytes}},
},
}},
}
params, err := buildOpenAIParams("fallback", req) // currently: unsupported content part
```
## Notes
- The OpenAI Go SDK already provides `responses.ResponseInputImageParam` and `responses.ResponseInputFileParam` (used by the Chat Completions→Responses conversion in the same dependency).
- adk-python already implements this exact behavior — `_inline_data_part_to_response_content` and `_file_data_part_to_response_content` in `src/google/adk/labs/openai/_openai_responses_llm.py`; per `AGENTS.md`, adk-python is the source of truth for feature behavior, so adk-go should align:
https://github.com/google/adk-python/blob/370027a770b413ab7991afa3395dbf8be1eb89e5/src/google/adk/labs/openai/_openai_responses_llm.py#L343-L380
- The existing `TestBuildOpenAIParams_UnsupportedPart` test would need to be updated (it currently asserts the error case).
Contributor guide
Assessment
This issue has not been assessed yet.