googleapis / googleapis/python-aiplatform
Allow mounting custom container on host's or external storage for online prediction
- Vorherrschende Sprache
- Python
- Sterne
- 905
- Forks
- 465
- Ø Merge
- 1 T. 13 Std.
- Gemergte PRs (30 T.)
- 44
Beschreibung
Thanks for stopping by to let us know something could be better!
**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.
**Is your feature request related to a problem? Please describe.**
When deploying a model using `aiplatform.Model.upload` `and aiplatform.Model.deploy` storage is limited to container's default,
so when using VLLM (e.g) with a big model that uses ray for cluster managment and there's memory spillage onto `/tmp/` directory, that directory fills up to the maximum and the model crashes..
can't go around it..
This happened to me when trying to deploy llama70b on 8L4 gpus with vllm
**Describe the solution you'd like**
Not sure about what's possible, but ultimately i'd like anothoer argument to the `upload` function that's something like "serving_container_tmp_dir_capacity_mb" or "serving_container_volume_mapping: list[dict[host_path,container_path]]" or "serving_container_mount_root_external_gcs_bucket: str (a gcs bucket that / is mounted on if possible)"
**Describe alternatives you've considered**
I've tried approaching the error from ray/vllm, setting the spill directory to be external, did not work..
**Additional context**
Add any other context or screenshots about the feature request here.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.