googleapis / googleapis/python-genai

Multimodal function responses do not work

Open
#2,268 1 comment 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

#### Environment details

- Programming language: Python
- OS: Linux 6.8.0-106-generic
- Language runtime version: 3.12.13
- Package version: 1.72.0

when i try and run the [multimodal function call ]( https://ai.google.dev/gemini-api/docs/function-calling?example=meeting#multimodal) example I get the following error

400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The referenced name `instrument.jpg` in function_response.response does not match to a display_name in the function_response.parts.', 'status': 'INVALID_ARGUMENT'}}

My code below - very close to the documented example I had to change
- added a default case to the function calling argument as the model was not providing an arg
- removed function calling id from the function call response (example is out of date as function calling ids no longer required)

from google import genai
from google.genai import types

import requests

client = genai.Client()

# This is a manual, two turn multimodal function calling workflow:

# 1. Define the function tool
get_image_declaration = types.FunctionDeclaration(
name="get_image",
description="Retrieves the image file reference for a specific order item.",
parameters={
"type": "object",
"properties": {
"item_name": {
"type": "string",
"description": "The name or description of the item ordered (e.g., 'instrument')."
}
},
"required": ["item_name"],
},
)
tool_config = types.Tool(function_declarations=[get_image_declaration])

# 2. Send a message that triggers the tool
prompt = "Show me the instrument I ordered last month."
response_1 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=[prompt],
config=types.GenerateContentConfig(
tools=[tool_config],
)
)

# 3. Handle the function call
function_call = response_1.function_calls[0]
requested_item = function_call.args.get("item_name", 'test')
print(f"Model wants to call: {function_call.name}")

# Execute your tool (e.g., call an API)
# (This is a mock response for the example)
print(f"Calling external tool for: {requested_item}")

function_response_data = {
"image_ref": {"$ref": "instrument.jpg"},
}

image_path = "https://goo.gle/instrument-img"
image_bytes = requests.get(image_path).content
function_response_multimodal_data = types.FunctionResponsePart(
inline_data=types.FunctionResponseBlob(
mime_type="image/jpeg",
display_name="instrument.jpg",
data=image_bytes,
)
)

# 4. Send the tool's result back
# Append this turn's messages to history for a final response.
history = [
types.Content(role="user", parts=[types.Part(text=prompt)]),
response_1.candidates[0].content,
types.Content(
role="user",
parts=[
types.Part.from_function_response(
name=function_call.name,
response=function_response_data,
parts=[function_response_multimodal_data]
)
],
)
]

response_2 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=history,
config=types.GenerateContentConfig(
tools=[tool_config],
thinking_config=types.ThinkingConfig(include_thoughts=True)
),
)

print(f"\nFinal model response: {response_2.text}")

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.