googleapis / googleapis/python-aiplatform

Allow mounting custom container on host's or external storage for online prediction

未关闭
#3,677 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**
When deploying a model using `aiplatform.Model.upload` `and aiplatform.Model.deploy` storage is limited to container's default,
so when using VLLM (e.g) with a big model that uses ray for cluster managment and there's memory spillage onto `/tmp/` directory, that directory fills up to the maximum and the model crashes..
can't go around it..

This happened to me when trying to deploy llama70b on 8L4 gpus with vllm
**Describe the solution you'd like**
Not sure about what's possible, but ultimately i'd like anothoer argument to the `upload` function that's something like "serving_container_tmp_dir_capacity_mb" or "serving_container_volume_mapping: list[dict[host_path,container_path]]" or "serving_container_mount_root_external_gcs_bucket: str (a gcs bucket that / is mounted on if possible)"

**Describe alternatives you've considered**
I've tried approaching the error from ray/vllm, setting the spill directory to be external, did not work..
**Additional context**
Add any other context or screenshots about the feature request here.

贡献指南

打开贡献指南

调研方向

从 aiplatform.Model.upload 和 aiplatform.Model.deploy 入口点开始,调查在线预测容器如何接收存储。完成的标准是定义一种受支持的方式,为已部署的容器提供额外的临时存储或外部存储,并对该行为进行文档说明且由相关测试覆盖。

由索引模型根据 Issue 内容生成。

评估

技术栈
gcp, google-cloud, python
领域
cloud, machine-learning
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。