AI-Hypercomputer / AI-Hypercomputer/tpu-recipes
Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe
- Lingua principale
- Shell
- Stelle
- 126
- Fork
- 81
- Merge medio
- 1g 15h
- PR unite (30g)
- 3
Descrizione
Hey team,
I followed the setup provided by the [TPU7X GPT-OSS vLLM inference recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:
```
RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.
```
Run Details:
- Manifests: [[Link](http://paste.googleplex.com/5740430492827648)]
- Error Log: [[Link](https://cloudlogging.app.goo.gl/879kfQ783FMkY7p48)]
- Full Logs: [[Link](https://cloudlogging.app.goo.gl/LkbcsipcV3mkPvus5)]
I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.
I would like to better understand:
- How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
- The performance impacts of lowering gpu-memory-utilization.
- Whether there are better mitigation strategies for this issue.
Thanks for your help!
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.