huggingface / huggingface/diffusers

flux:different results on different machines.

Aperta
#10,075 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

Hi!
During the debugging phase, I've noticed that training the [FLUX-Controlnet-Inpainting](https://github.com/alimama-creative/FLUX-Controlnet-Inpainting) model yields different results on different machines.

Specifically, on one machine, everything trains fine,
but on another, I'm getting an error that says "Gradient for module.xxx contains NaN!".

I've double-checked and confirmed that my code is exactly the same on both machines, and use the same seed in set_seed().

Has anyone else encountered this same issue? I'm curious to know if there are any potential causes for this discrepancy in training results across different machines. Any insights or suggestions would be greatly appreciated.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the linked FLUX-Controlnet-Inpainting training setup and compare the two machines, including the environment and the point where the NaN gradient appears. Reproduce the discrepancy with the same seed; done means identifying a machine-dependent cause or a reliably reproducible failure and documenting the relevant conditions.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.