googleapis / googleapis/python-genai

Multimodal function responses do not work

Aperta
#2,268 1 commento 0 reazioni 1 assegnatario Rivendicata da @Venkaiahbabuneelam Vedi su GitHub
priority: p2 type: bug
Lingua principale
Python
Stelle
4k
Fork
1k
Merge medio
2g 12h
PR unite (30g)
41

Descrizione

#### Environment details

- Programming language: Python
- OS: Linux 6.8.0-106-generic
- Language runtime version: 3.12.13
- Package version: 1.72.0

when i try and run the [multimodal function call ]( https://ai.google.dev/gemini-api/docs/function-calling?example=meeting#multimodal) example I get the following error

400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The referenced name `instrument.jpg` in function_response.response does not match to a display_name in the function_response.parts.', 'status': 'INVALID_ARGUMENT'}}

My code below - very close to the documented example I had to change
- added a default case to the function calling argument as the model was not providing an arg
- removed function calling id from the function call response (example is out of date as function calling ids no longer required)

from google import genai
from google.genai import types

import requests

client = genai.Client()

# This is a manual, two turn multimodal function calling workflow:

# 1. Define the function tool
get_image_declaration = types.FunctionDeclaration(
name="get_image",
description="Retrieves the image file reference for a specific order item.",
parameters={
"type": "object",
"properties": {
"item_name": {
"type": "string",
"description": "The name or description of the item ordered (e.g., 'instrument')."
}
},
"required": ["item_name"],
},
)
tool_config = types.Tool(function_declarations=[get_image_declaration])

# 2. Send a message that triggers the tool
prompt = "Show me the instrument I ordered last month."
response_1 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=[prompt],
config=types.GenerateContentConfig(
tools=[tool_config],
)
)

# 3. Handle the function call
function_call = response_1.function_calls[0]
requested_item = function_call.args.get("item_name", 'test')
print(f"Model wants to call: {function_call.name}")

# Execute your tool (e.g., call an API)
# (This is a mock response for the example)
print(f"Calling external tool for: {requested_item}")

function_response_data = {
"image_ref": {"$ref": "instrument.jpg"},
}

image_path = "https://goo.gle/instrument-img"
image_bytes = requests.get(image_path).content
function_response_multimodal_data = types.FunctionResponsePart(
inline_data=types.FunctionResponseBlob(
mime_type="image/jpeg",
display_name="instrument.jpg",
data=image_bytes,
)
)

# 4. Send the tool's result back
# Append this turn's messages to history for a final response.
history = [
types.Content(role="user", parts=[types.Part(text=prompt)]),
response_1.candidates[0].content,
types.Content(
role="user",
parts=[
types.Part.from_function_response(
name=function_call.name,
response=function_response_data,
parts=[function_response_multimodal_data]
)
],
)
]

response_2 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=history,
config=types.GenerateContentConfig(
tools=[tool_config],
thinking_config=types.ThinkingConfig(include_thoughts=True)
),
)

print(f"\nFinal model response: {response_2.text}")

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.