googleapis / googleapis/python-aiplatform

Endpoint prediction GRPC error

Open
#1,849 5 comments 0 reactions 1 assignee Claimed by @kthytang View on GitHub
api: vertex-ai
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

Getting a hard to debug error when attempting to send predictions to deployed Endpoint. The endpoint is serving predictions just fine via `curl` but fails when using the client library. Other methods on the endpoint work fine, just not prediction it seems.

#### Environment details

- OS type and version: Ubuntu 20.04.5 LTS
- Python version: 3.9.7
- pip version: 22.3
- `google-cloud-aiplatform` version: 1.19.1

#### Steps to reproduce

1. Create Vertex Endpoint for custom trained model
2. Send prediction request

#### Code example

```python
aiplatform.init(project="my-project", location="my-location")
endpoint = aiplatform.Endpoint("endpoint-id")
predictions = endpoint.predict(instances=[{...}, {...}, {...}], parameters={})
print(predictions)
```

#### Stack trace
```
Traceback (most recent call last):
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 2"
debug_error_string = "UNKNOWN:Error received from peer ipv4:***:443 {grpc_message:"Received RST_STREAM with error code 2", grpc_status:13, created_time:"2022-12-13T18:05:21.914025462+00:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/jupyter/***/***-model/predict.py", line 12, in
predictions = endpoint.predict(instances=input_dict["instances"], parameters=input_dict["parameters"])
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1503, in predict
prediction_response = self._prediction_client.predict(
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 589, in predict
response = rpc(
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "/home/jupyter/.pyenv/versions/***/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 2
```

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.