AI-Hypercomputer / AI-Hypercomputer/xpk

[slice] Reschedule after Slice Failed

Aberta
#685 3 comentários 0 reações 1 responsável Reivindicada por @pajakd Ver no GitHub
Linguagem predominante
Python
Estrelas
193
Forks
94
Merge médio
24min
PRs com merge (30d)
1

Descrição

When testing the super-slice feature manually:
1. Create Jobset
2. (Slice CR gets created)
3. Transition the Slice state to Ready
4. Workload gets admitted
5. Transition the slice state to Error
6. Workload get suspended
7. (after ~1 minute) Slice CR is garbage-collected

The Workload is never unsuspended, the new Slice CR object is not created.

The reason for this is that we transition the admission check state to Rejected:
https://github.com/AI-Hypercomputer/xpk/blob/34c7fc706b1c2a4320457b040c18a9d8c9edf03d/slice/internal/controller/workload_controller.go#L619-L620

Which is not retried:

https://github.com/kubernetes-sigs/kueue/blob/6a1f89a58334b282f0c820b889d4137a4bdd6249/apis/kueue/v1beta1/admissioncheck_types.go#L32-L35

I think we should in this case transition to `CheckStateRetry` to
A) Give the slice some time to recover
B) Create a new slice if the old one does not recover after it is deleted

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.