AI-Hypercomputer / AI-Hypercomputer/ray-tpu

Maybe fork a separate process to monitor execution

Abierto
#13 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
15
Forks
8
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

A main use case for Ray-on-TPU is for orchestrating JAX training jobs. XLA likes to abort the process when things go wrong, whichh ends the process outside of Python control flow and means that Ray can't tell the difference between the Ray process crashing due to some internal Ray error or XLA killing it. To help with retry logic, we have found it useful to fork a process to run the main payload function.

We're happy to keep this in our library if you don't think it's necessary. Just raising it for now.

Our implementation and justification: https://github.com/stanford-crfm/levanter/blob/94afdc17f6091249e70e90cb27b4378d0553ff56/src/levanter/infra/ray_tpu.py#L459

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.