lightly-ai / lightly-ai/lightly
Add SimCLR with temperature schedule
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.8k
- Forks
- 367
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 5
Description
https://arxiv.org/abs/2303.13664 shows that varying the temperature in the contrastive loss during pretraining can help models learn better representations for datasets with long-tail distributions and doesn't hurt performance for datasets with uniform distributions. This could be an interesting method to add to the package.
I would start with running a benchmark on a normal dataset (ImageNet) as we don't have standard long-tail datasets in our benchmarks. It would also be interesting to see if it also gives good performance on full ImageNet as the paper only reports results for ImageNet100-LT. If it also works on default ImageNet it could be a good default method to add to most contrastive models.
TODO:
- [ ] Add optional period to [`cosine_schedule`](https://github.com/lightly-ai/lightly/blob/df004d288fa07816e55975fa0af92868f9867436/lightly/utils/scheduler.py#L7)
- [ ] Add benchmark with SimCLR and temperature schedule
- [ ] Run benchmark and report results
- [ ] Add example and docs
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with lightly/utils/scheduler.py and the linked paper, then inspect how SimCLR benchmarks are organized in the package. Evaluate the temperature schedule on ImageNet, report the benchmark results, and review the existing example and documentation structure. Done means the optional cosine-schedule period, benchmark, results, example, and docs are all addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100