googleapis / googleapis/python-aiplatform

Allow mounting custom container on host's or external storage for online prediction

Offen
#3,677 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
api: vertex-ai
Vorherrschende Sprache
Python
Sterne
905
Forks
465
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
44

Beschreibung

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**
When deploying a model using `aiplatform.Model.upload` `and aiplatform.Model.deploy` storage is limited to container's default,
so when using VLLM (e.g) with a big model that uses ray for cluster managment and there's memory spillage onto `/tmp/` directory, that directory fills up to the maximum and the model crashes..
can't go around it..

This happened to me when trying to deploy llama70b on 8L4 gpus with vllm
**Describe the solution you'd like**
Not sure about what's possible, but ultimately i'd like anothoer argument to the `upload` function that's something like "serving_container_tmp_dir_capacity_mb" or "serving_container_volume_mapping: list[dict[host_path,container_path]]" or "serving_container_mount_root_external_gcs_bucket: str (a gcs bucket that / is mounted on if possible)"

**Describe alternatives you've considered**
I've tried approaching the error from ray/vllm, setting the spill directory to be external, did not work..
**Additional context**
Add any other context or screenshots about the feature request here.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.