AI-Hypercomputer / AI-Hypercomputer/tpu-recipes
Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe
- Lenguaje dominante
- Shell
- Estrellas
- 126
- Forks
- 81
- Merge medio
- 1 d 15 h
- PR fusionados (30 d)
- 3
Descripción
Hey team,
I followed the setup provided by the [TPU7X GPT-OSS vLLM inference recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:
```
RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.
```
Run Details:
- Manifests: [[Link](http://paste.googleplex.com/5740430492827648)]
- Error Log: [[Link](https://cloudlogging.app.goo.gl/879kfQ783FMkY7p48)]
- Full Logs: [[Link](https://cloudlogging.app.goo.gl/LkbcsipcV3mkPvus5)]
I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.
I would like to better understand:
- How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
- The performance impacts of lowering gpu-memory-utilization.
- Whether there are better mitigation strategies for this issue.
Thanks for your help!
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.