huggingface / huggingface/diffusers

flux:different results on different machines.

Offen
#10,075 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

Hi!
During the debugging phase, I've noticed that training the [FLUX-Controlnet-Inpainting](https://github.com/alimama-creative/FLUX-Controlnet-Inpainting) model yields different results on different machines.

Specifically, on one machine, everything trains fine,
but on another, I'm getting an error that says "Gradient for module.xxx contains NaN!".

I've double-checked and confirmed that my code is exactly the same on both machines, and use the same seed in set_seed().

Has anyone else encountered this same issue? I'm curious to know if there are any potential causes for this discrepancy in training results across different machines. Any insights or suggestions would be greatly appreciated.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with the linked FLUX-Controlnet-Inpainting training setup and compare the two machines, including the environment and the point where the NaN gradient appears. Reproduce the discrepancy with the same seed; done means identifying a machine-dependent cause or a reliably reproducible failure and documenting the relevant conditions.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.