AI-Hypercomputer / AI-Hypercomputer/tpu-recipes

Frequently got `resource exhausted error` when start up the server based on GPT-OSS vLLM inference recipe

Abierto
#151 4 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Shell
Estrellas
126
Forks
81
Merge medio
1 d 15 h
PR fusionados (30 d)
3

Descripción

Hey team,

I followed the setup provided by the [TPU7X GPT-OSS vLLM inference recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) but frequently (63.64% error rate) encountered a RESOURCE_EXHAUSTED error when starting the inference server:

```
RESOURCE_EXHAUSTED: Error loading program 'jit__multi_slice': Attempting to reserve 6.14G at the bottom of memory. That was not possible.
```

Run Details:
- Manifests: [[Link](http://paste.googleplex.com/5740430492827648)]
- Error Log: [[Link](https://cloudlogging.app.goo.gl/879kfQ783FMkY7p48)]
- Full Logs: [[Link](https://cloudlogging.app.goo.gl/LkbcsipcV3mkPvus5)]

I found that reducing the gpu-memory-utilization value from 0.93 to 0.83 allows the server to start as expected.

I would like to better understand:

- How gpu-memory-utilization impacts specific memory allocation and why this error occurs intermittently.
- The performance impacts of lowering gpu-memory-utilization.
- Whether there are better mitigation strategies for this issue.

Thanks for your help!

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.