AI-Hypercomputer / AI-Hypercomputer/tpu-recipes

Parallelism setups might not correct in OPT-OSS 120B vLLM inference recipes

未關閉
#161 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Shell
星號
126
分支
81
平均合併
1 天 15 小時
30 天內合併 PR
3

描述

When I ran the benchmark following these recipes:
- https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/GPT-OSS/README.md
- https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/GPT-OSS/README-gcs.md

I found that only one model copy was downloaded to the node from GCS, even though tp=2 was set.

Upon further inspection of the logs, I noticed that data_parallelism switches from [4](https://cloudlogging.app.goo.gl/ZH9WzYWzvPhv1weZ7) to [1](https://cloudlogging.app.goo.gl/wyq1UhchSzi6MiWN9) in the vLLM engine.

I would like to understand this behavior better. Specifically, based on the current configurations, how many model copies should reside on a single node, and how many copies are actually downloaded from GCS? Does the workload download the model multiple times, or is it downloaded once and then shared across the chips?

cc @karan

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。