googleapis / googleapis/python-genai

Validating batch response with Google Search grounding against `GenerateContentResponse` causes validation error

Open
#2,510 1 comment 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

It seems like the `GenerateContentResponse` objects and its nested objects are not designed properly for the any reponses that uses the built-in Google Search tool.

It returns:
```
response.candidates.0.groundingMetadata.groundingChunks.0.chunkMetadata
Extra inputs are not permitted [type=extra_forbidden, input_value={'query': 'query'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/extra_forbidden
```

This is caused by the `GroundingChunk` object, which does not actually match the `groundingChunk` returned by the Vertex AI API:
```python
class GroundingMetadata(_common.BaseModel):
"""Information for various kinds of grounding."""

grounding_chunks: Optional[list[GroundingChunk]] = Field(
default=None,
description="""A list of supporting references retrieved from the grounding
source. This field is populated when the grounding source is Google
Search, Vertex AI Search, or Google Maps.
""",
)
# Truncated for brevity

class GroundingChunk(_common.BaseModel):
"""A piece of evidence that supports a claim made by the model.

This is used to show a citation for a claim made by the model. When grounding
is enabled, the model returns a `GroundingChunk` that contains a reference to
the source of the information.
"""

image: Optional[GroundingChunkImage] = Field(
default=None,
description="""A grounding chunk from an image search result. See the `Image` message for details.""",
)
maps: Optional[GroundingChunkMaps] = Field(
default=None,
description="""A `Maps` chunk is a piece of evidence that comes from Google Maps.

It contains information about a place, such as its name, address, and
reviews. This is used to provide the user with rich, location-based
information.""",
)
retrieved_context: Optional[GroundingChunkRetrievedContext] = Field(
default=None,
description="""A grounding chunk from a data source retrieved by a retrieval tool, such as Vertex AI Search. See the `RetrievedContext` message for details""",
)
web: Optional[GroundingChunkWeb] = Field(
default=None,
description="""A grounding chunk from a web page, typically from Google Search. See the `Web` message for details.""",
)

```

The actual JSON `groundingChunk` JSON:

```json
"groundingMetadata" : {
"groundingChunks" : [ {
"chunkMetadata" : {
"query" : "dummy_query"
},
"web" : {
"title" : "youtube.com",
"uri" : "https://vertexaisearch.cloud.google.com/grounding-api-redirect/id"
}
} ],
```

#### Environment details

- Programming language: Python
- OS: MacOS
- Language runtime version: 3.12
- Package version: 2.6.0

#### Steps to reproduce

1. Create batch request or requests with google search enabled and ensure the model actually performs a Google search , dummy example:
```json
{
"contents" : [ {
"parts" : [ {
"text" : "Dummy request"
} ],
"role" : "user"
} ],
"generationConfig" : {
"responseMimeType" : "application/json",
"responseSchema" : {
},
"temperature" : 0.2
},
"systemInstruction" : {
"parts" : [ {
"text" : "Dummy system instructions "
} ]
},
"tools" : [ {
"googleSearch" : {
"excludeDomains" : [ "example.com" ] // google search on batch inference. does not work unless you actually specify an exclusion domain... please fix as well
}
} ]
}
```
2. After results are ready, read the results and validate the `GenerationResponse` object againts the "response" part of the JSON response:
```python
from google.genai import types
result = types.GenerateContentResponse.model_validate_json(parsed_json_dict["response"]) # Raises validation error
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.