lightly-ai / lightly-ai/lightly
Divergent loss with SimCLR
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 367
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 5
Description
Sometimes, when training using the SimCLR method I get some divergent loss function (see attached screenshot). I wonder if anyone has ever experienced this kind of issue when training with SimCLR. This has happened to me on different occasions with ResNet-18/50 model.
I don't think that this is an issue with the code, but if anyone has ever seen this kind of problem I would be grateful for your input.
Here's some information about the training hyper-parameters:
- multi-gpu training with
batch_sizeof 256 per GPU batch_size: 1024criterion:NTXentLosswith temperature of 0.1 andgather_distributed=Trueoptimizer: LARS with a base learning rate of 0.3 and default parameters from herescheduler:CosineWarmupSchedulerwith 10k warmup steps
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with benchmarks/imagenet/resnet50/simclr.py, the linked reference for default LARS settings, and the NTXentLoss/CosineWarmupScheduler configuration described here. Reproduce the reported multi-GPU run with the stated batch sizes, temperature, learning rate, and 10k warmup; done means isolating whether the divergence is reproducible and documenting a confirmed cause or next action.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100