googleapis / googleapis/python-aiplatform
Endpoint prediction GRPC error
- Dominant language
- Python
- Stars
- 905
- Forks
- 465
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Description
Getting a hard to debug error when attempting to send predictions to deployed Endpoint. The endpoint is serving predictions just fine via `curl` but fails when using the client library. Other methods on the endpoint work fine, just not prediction it seems.
#### Environment details
- OS type and version: Ubuntu 20.04.5 LTS
- Python version: 3.9.7
- pip version: 22.3
- `google-cloud-aiplatform` version: 1.19.1
#### Steps to reproduce
1. Create Vertex Endpoint for custom trained model
2. Send prediction request
#### Code example
```python
aiplatform.init(project="my-project", location="my-location")
endpoint = aiplatform.Endpoint("endpoint-id")
predictions = endpoint.predict(instances=[{...}, {...}, {...}], parameters={})
print(predictions)
```
#### Stack trace
```
Traceback (most recent call last):
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 2"
debug_error_string = "UNKNOWN:Error received from peer ipv4:***:443 {grpc_message:"Received RST_STREAM with error code 2", grpc_status:13, created_time:"2022-12-13T18:05:21.914025462+00:00"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/jupyter/***/***-model/predict.py", line 12, in
predictions = endpoint.predict(instances=input_dict["instances"], parameters=input_dict["parameters"])
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1503, in predict
prediction_response = self._prediction_client.predict(
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 589, in predict
response = rpc(
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 2
```
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.