AI-Hypercomputer / AI-Hypercomputer/tpu-recipes

Parallelism setups might not correct in OPT-OSS 120B vLLM inference recipes

Offen
#161 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Shell
Sterne
126
Forks
81
Ø Merge
1 T. 15 Std.
Gemergte PRs (30 T.)
3

Beschreibung

When I ran the benchmark following these recipes:
- https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/GPT-OSS/README.md
- https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/GPT-OSS/README-gcs.md

I found that only one model copy was downloaded to the node from GCS, even though tp=2 was set.

Upon further inspection of the logs, I noticed that data_parallelism switches from [4](https://cloudlogging.app.goo.gl/ZH9WzYWzvPhv1weZ7) to [1](https://cloudlogging.app.goo.gl/wyq1UhchSzi6MiWN9) in the vLLM engine.

I would like to understand this behavior better. Specifically, based on the current configurations, how many model copies should reside on a single node, and how many copies are actually downloaded from GCS? Does the workload download the model multiple times, or is it downloaded once and then shared across the chips?

cc @karan

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.