googleapis / googleapis/python-genai

Multimodal function responses do not work

Offen
#2,268 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @Venkaiahbabuneelam Auf GitHub ansehen
priority: p2 type: bug
Vorherrschende Sprache
Python
Sterne
4k
Forks
1k
Ø Merge
2 T. 12 Std.
Gemergte PRs (30 T.)
41

Beschreibung

#### Environment details

- Programming language: Python
- OS: Linux 6.8.0-106-generic
- Language runtime version: 3.12.13
- Package version: 1.72.0

when i try and run the [multimodal function call ]( https://ai.google.dev/gemini-api/docs/function-calling?example=meeting#multimodal) example I get the following error

400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'The referenced name `instrument.jpg` in function_response.response does not match to a display_name in the function_response.parts.', 'status': 'INVALID_ARGUMENT'}}

My code below - very close to the documented example I had to change
- added a default case to the function calling argument as the model was not providing an arg
- removed function calling id from the function call response (example is out of date as function calling ids no longer required)

from google import genai
from google.genai import types

import requests

client = genai.Client()

# This is a manual, two turn multimodal function calling workflow:

# 1. Define the function tool
get_image_declaration = types.FunctionDeclaration(
name="get_image",
description="Retrieves the image file reference for a specific order item.",
parameters={
"type": "object",
"properties": {
"item_name": {
"type": "string",
"description": "The name or description of the item ordered (e.g., 'instrument')."
}
},
"required": ["item_name"],
},
)
tool_config = types.Tool(function_declarations=[get_image_declaration])

# 2. Send a message that triggers the tool
prompt = "Show me the instrument I ordered last month."
response_1 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=[prompt],
config=types.GenerateContentConfig(
tools=[tool_config],
)
)

# 3. Handle the function call
function_call = response_1.function_calls[0]
requested_item = function_call.args.get("item_name", 'test')
print(f"Model wants to call: {function_call.name}")

# Execute your tool (e.g., call an API)
# (This is a mock response for the example)
print(f"Calling external tool for: {requested_item}")

function_response_data = {
"image_ref": {"$ref": "instrument.jpg"},
}

image_path = "https://goo.gle/instrument-img"
image_bytes = requests.get(image_path).content
function_response_multimodal_data = types.FunctionResponsePart(
inline_data=types.FunctionResponseBlob(
mime_type="image/jpeg",
display_name="instrument.jpg",
data=image_bytes,
)
)

# 4. Send the tool's result back
# Append this turn's messages to history for a final response.
history = [
types.Content(role="user", parts=[types.Part(text=prompt)]),
response_1.candidates[0].content,
types.Content(
role="user",
parts=[
types.Part.from_function_response(
name=function_call.name,
response=function_response_data,
parts=[function_response_multimodal_data]
)
],
)
]

response_2 = client.models.generate_content(
model="gemini-3-flash-preview",
contents=history,
config=types.GenerateContentConfig(
tools=[tool_config],
thinking_config=types.ThinkingConfig(include_thoughts=True)
),
)

print(f"\nFinal model response: {response_2.text}")

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.