googleapis / googleapis/python-aiplatform

Model deploy timeout in 900 seconds using Vertex AI Python SDK.

Đang mở
#3,329 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: vertex-ai
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

#### Environment details

- OS type and version: `MacOS 14.1.1 (23B2082)`
- Python version: `Python 3.9.6`
- pip version: `pip 21.2.4`
- google-cloud-aiplatform version: `1.40.0`

#### Steps to reproduce
I am using Vertex AI Python SDK to deploy `Mistral/Mistral-7B-v0.1` on GCP by running the following example code on [Google Colab notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb). The model deployment script times out in 900 seconds, even though the parameter `deploy_request_timeout` is set to `1800`. It doesn't seem to propagate to google api-core. The model sometimes takes longer to get to `Ready` state, but the script times out early.

#### Code example

```python
model.deploy(
endpoint=endpoint,
machine_type=machine_type,
accelerator_type=accelerator_type,
accelerator_count=accelerator_count,
deploy_request_timeout=1800,
service_account=service_account,
enable_access_logging=True
)
```

#### Stack trace
```
Traceback (most recent call last):
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
result = target()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 120, in _done_or_raise
raise _OperationNotComplete()
google.api_core.future.polling._OperationNotComplete

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll
polling(self._done_or_raise)(retry=retry)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
return retry_target(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
_retry_error_helper(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 221, in _retry_error_helper
raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 900.0s exceeded, last exception:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ananyaraval/workspace/gcp-tools/launch_vertex.py", line 154, in
}]
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1557, in predict
self.wait()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 260, in wait
self._raise_future_exception()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 226, in _raise_future_exception
raise self._exception
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 238, in _complete_future
future.result() # raises
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 328, in wait_for_dependencies_and_invoke
result = method(*args, **kwargs)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 3750, in _deploy
endpoint._deploy_call(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1300, in _deploy_call
operation_future.result(timeout=None)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 256, in result
self._blocking_poll(timeout=timeout, retry=retry, polling=polling)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 139, in _blocking_poll
raise concurrent.futures.TimeoutError(
concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout of 900 seconds.
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.