Unexpected InvalidArgument error for large response_schema

I’ve encountered an issue with the Gemini 1.5 Pro API where there seems to be an undocumented size limit for the response_schema parameter in GenerationConfig. When attempting to use a schema with a large number of properties or long property names, the API throws an InvalidArgument error without further details.

#### Environment details

- OS type and version: Ubuntu Linux 24.04 / macOS 14.6.1
- Python version: 3.12.5
- pip version: 24.0
- `google-cloud-aiplatform` version: 1.63.0

#### Steps to reproduce

1. Run the below code with ADC set up
2. Observe InvalidArgument exception w/o further details
3. Change the parameters and re-run the case. There's a sweet spot to the number of schema keys and their string length:
- Normal response:
- `num_properties` = 42
- `property_name_length` = 34
- InvalidArgument:
- `num_properties` = 43
- `property_name_length` = 34
- InvalidArgument:
- `num_properties` = 42
- `property_name_length` = 35

#### Code example

```python
import copy
from pprint import pprint
import random
import string
from vertexai.generative_models import GenerationConfig, GenerativeModel
from google.api_core.exceptions import InvalidArgument

prompt = "Respond according to the JSON schema."
num_properties = 40
property_name_length = 40

properties = [
"".join(random.choices(string.ascii_lowercase, k=property_name_length))
for _ in range(num_properties)
]

json_schema = {
"type": "object",
"properties": {name: {"type": "string"} for name in properties},
}

model = GenerativeModel("gemini-1.5-pro-001")

try:
response = await model.generate_content_async(
contents=prompt,
generation_config=GenerationConfig(
temperature=0.0,
response_mime_type="application/json",
response_schema=copy.deepcopy(json_schema),
),
stream=True,
)
except InvalidArgument as e:
print("Request failed as expected with InvalidArgument error:")
print(e)
print(
f"generation_config.response_schema had {num_properties} properties, {property_name_length} characters each:"
)
pprint(json_schema)
```

#### Stack trace

```
---------------------------------------------------------------------------
AioRpcError Traceback (most recent call last)
File .venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:77, in _WrappedCall.wait_for_connection(self)
76 try:
---> 77 await self._call.wait_for_connection()
78 except grpc.RpcError as rpc_error:

File .venv/lib/python3.12/site-packages/grpc/aio/_call.py:650, in UnaryStreamCall.wait_for_connection(self)
649 if self.done():
--> 650 await self._raise_for_status()

File .venv/lib/python3.12/site-packages/grpc/aio/_call.py:263, in Call._raise_for_status(self)
262 if code != grpc.StatusCode.OK:
--> 263 raise _create_rpc_error(
264 await self.initial_metadata(), await self._cython_call.status()
265 )

AioRpcError:

The above exception was the direct cause of the following exception:
…
77 await self._call.wait_for_connection()
78 except grpc.RpcError as rpc_error:
---> 79 raise exceptions.from_grpc_error(rpc_error) from rpc_error

InvalidArgument: 400 Request contains an invalid argument.
```

Note that I've actually also received a 500 AioRpcError response.

---

Is there an internal size limit on the in-memory size of a response_schema? If so, should there be? Is there any documentation on it?

(I'm aware this is likely not an issue with the Python API client but the service itself but hey, the API client is generated from a service description sooo… :D)