AI-Hypercomputer / AI-Hypercomputer/tpu-recipes

Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe

Aperta
#151 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Shell
Stelle
126
Fork
81
Merge medio
1g 15h
PR unite (30g)
3

Descrizione

Hey team,

I followed the setup provided by the [TPU7X GPT-OSS vLLM inference recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:

```
RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.
```

Run Details:
- Manifests: [[Link](http://paste.googleplex.com/5740430492827648)]
- Error Log: [[Link](https://cloudlogging.app.goo.gl/879kfQ783FMkY7p48)]
- Full Logs: [[Link](https://cloudlogging.app.goo.gl/LkbcsipcV3mkPvus5)]

I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.

I would like to better understand:

- How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
- The performance impacts of lowering gpu-memory-utilization.
- Whether there are better mitigation strategies for this issue.

Thanks for your help!

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.