lllyasviel / lllyasviel/ControlNet
Training loss unconverged, output unmatched with input
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 34.1k
- Forks
- 3k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thanks for this great work!
We try to train the model to inpaint, followed your instructions to generate prompts from BLIP and accumulate grad. Now we use a batchsize of 4 x grad accumulation of 16, but the loss just fluctuated over time and never went down. This is the loss curve:
And as a result, the output does not match the input part either. It has ok quality on its own, though.
I was considering,
- if I modify the DDPM pipeline, the output result(from noise, sample 50 DDIM steps) could be more aligned with input, do you have any suggestions? Also,
- it still does not solve the training loss problem where time steps are randomly sampled.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported inpainting training setup: batch size 4, gradient accumulation 16, BLIP-generated prompts, and randomly sampled timesteps. Inspect the DDPM pipeline and 50-step DDIM sampling behavior, then compare training loss and input/output alignment against the report. Done means the cause of the fluctuating loss and unmatched input is identified and a validated correction is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100