google / google/adk-go

[Feature]: model/openaimodel: support image/file input (genai InlineData/FileData parts)

Open
#1,333 7 comments 2 reactions 1 assignee Claimed by @baptmont View on GitHub
enhancement needs review
Dominant language
Go
Stars
8.8k
Forks
1k
Avg merge
3d 18h
Merged PRs (30d)
88

Description

## Problem

The OpenAI model adapter (`model/openaimodel`) only supports **text** content parts. When a user passes an image or file (e.g. `genai.Part.InlineData` / `genai.Part.FileData`) to an agent using an OpenAI model, `convertContents` falls through to the `default` branch and returns:

```
openai: unsupported content part *genai.Part
```

Relevant code: `model/openaimodel/request.go:116` — the part switch in `convertContents` handles only `Text`, `FunctionCall`, and `FunctionResponse`. The behavior is even pinned by a test asserting failure: `TestBuildOpenAIParams_UnsupportedPart` (`model/openaimodel/request_test.go:123`).

## Expected behavior

Support OpenAI-format image/file input in the Responses API:

- `genai.Part.InlineData` (`*genai.Blob` with `Data` + `MimeType`) → `responses.ResponseInputImageParam` (`input_image`, base64 `data:...` or `file_id`) — covers images, and audio/video via `input_audio`/`input_video` where applicable.
- `genai.Part.FileData` (remote `URI` + `MimeType`) → `responses.ResponseInputFileParam` (`input_file`), e.g. PDFs.
- A message containing both text and image parts should be emitted as a single message with a content list mixing `input_text` and `input_image` items (the existing `newMessage` only builds `ResponseInputTextParam`).

## Repro sketch

```go
req := &model.LLMRequest{
Contents: []*genai.Content{{
Role: string(genai.RoleUser),
Parts: []*genai.Part{
{Text: "What is in this image?"},
{InlineData: &genai.Blob{MimeType: "image/png", Data: pngBytes}},
},
}},
}
params, err := buildOpenAIParams("fallback", req) // currently: unsupported content part
```

## Notes

- The OpenAI Go SDK already provides `responses.ResponseInputImageParam` and `responses.ResponseInputFileParam` (used by the Chat Completions→Responses conversion in the same dependency).
- adk-python already implements this exact behavior — `_inline_data_part_to_response_content` and `_file_data_part_to_response_content` in `src/google/adk/labs/openai/_openai_responses_llm.py`; per `AGENTS.md`, adk-python is the source of truth for feature behavior, so adk-go should align:

https://github.com/google/adk-python/blob/370027a770b413ab7991afa3395dbf8be1eb89e5/src/google/adk/labs/openai/_openai_responses_llm.py#L343-L380

- The existing `TestBuildOpenAIParams_UnsupportedPart` test would need to be updated (it currently asserts the error case).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.