AI-Hypercomputer / AI-Hypercomputer/tpu-recipes

Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe

Offen
#151 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Shell
Sterne
126
Forks
81
Ø Merge
1 T. 15 Std.
Gemergte PRs (30 T.)
3

Beschreibung

Hey team,

I followed the setup provided by the [TPU7X GPT-OSS vLLM inference recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:

```
RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.
```

Run Details:
- Manifests: [[Link](http://paste.googleplex.com/5740430492827648)]
- Error Log: [[Link](https://cloudlogging.app.goo.gl/879kfQ783FMkY7p48)]
- Full Logs: [[Link](https://cloudlogging.app.goo.gl/LkbcsipcV3mkPvus5)]

I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.

I would like to better understand:

- How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
- The performance impacts of lowering gpu-memory-utilization.
- Whether there are better mitigation strategies for this issue.

Thanks for your help!

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.