googleapis / googleapis/python-aiplatform

Model deploy timeout in 900 seconds using Vertex AI Python SDK.

オープン
#3,329 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
api: vertex-ai
主要言語
Python
スター
905
フォーク
465
平均マージ
1日 13時間
マージ済み PR(30日)
44

説明

#### Environment details

- OS type and version: `MacOS 14.1.1 (23B2082)`
- Python version: `Python 3.9.6`
- pip version: `pip 21.2.4`
- google-cloud-aiplatform version: `1.40.0`

#### Steps to reproduce
I am using Vertex AI Python SDK to deploy `Mistral/Mistral-7B-v0.1` on GCP by running the following example code on [Google Colab notebook](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb). The model deployment script times out in 900 seconds, even though the parameter `deploy_request_timeout` is set to `1800`. It doesn't seem to propagate to google api-core. The model sometimes takes longer to get to `Ready` state, but the script times out early.

#### Code example

```python
model.deploy(
endpoint=endpoint,
machine_type=machine_type,
accelerator_type=accelerator_type,
accelerator_count=accelerator_count,
deploy_request_timeout=1800,
service_account=service_account,
enable_access_logging=True
)
```

#### Stack trace
```
Traceback (most recent call last):
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
result = target()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 120, in _done_or_raise
raise _OperationNotComplete()
google.api_core.future.polling._OperationNotComplete

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll
polling(self._done_or_raise)(retry=retry)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
return retry_target(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
_retry_error_helper(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 221, in _retry_error_helper
raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 900.0s exceeded, last exception:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ananyaraval/workspace/gcp-tools/launch_vertex.py", line 154, in
}]
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1557, in predict
self.wait()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 260, in wait
self._raise_future_exception()
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 226, in _raise_future_exception
raise self._exception
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 238, in _complete_future
future.result() # raises
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/base.py", line 328, in wait_for_dependencies_and_invoke
result = method(*args, **kwargs)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 3750, in _deploy
endpoint._deploy_call(
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/cloud/aiplatform/models.py", line 1300, in _deploy_call
operation_future.result(timeout=None)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 256, in result
self._blocking_poll(timeout=timeout, retry=retry, polling=polling)
File "/Users/ananyaraval/workspace/unbias/unbias_pypi/lib/python3.9/site-packages/google/api_core/future/polling.py", line 139, in _blocking_poll
raise concurrent.futures.TimeoutError(
concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout of 900 seconds.
```

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。