googleapis / googleapis/python-aiplatform

Allow mounting custom container on host's or external storage for online prediction

オープン
#3,677 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
api: vertex-ai
主要言語
Python
スター
905
フォーク
465
平均マージ
1日 13時間
マージ済み PR(30日)
44

説明

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**
When deploying a model using `aiplatform.Model.upload` `and aiplatform.Model.deploy` storage is limited to container's default,
so when using VLLM (e.g) with a big model that uses ray for cluster managment and there's memory spillage onto `/tmp/` directory, that directory fills up to the maximum and the model crashes..
can't go around it..

This happened to me when trying to deploy llama70b on 8L4 gpus with vllm
**Describe the solution you'd like**
Not sure about what's possible, but ultimately i'd like anothoer argument to the `upload` function that's something like "serving_container_tmp_dir_capacity_mb" or "serving_container_volume_mapping: list[dict[host_path,container_path]]" or "serving_container_mount_root_external_gcs_bucket: str (a gcs bucket that / is mounted on if possible)"

**Describe alternatives you've considered**
I've tried approaching the error from ray/vllm, setting the spill directory to be external, did not work..
**Additional context**
Add any other context or screenshots about the feature request here.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。