lightly-ai / lightly-ai/lightly

Add SimCLR with temperature schedule

Open
#1,319 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Python
Stars
3.8k
Forks
367
Avg merge
3d 22h
Merged PRs (30d)
5

Description

https://arxiv.org/abs/2303.13664 shows that varying the temperature in the contrastive loss during pretraining can help models learn better representations for datasets with long-tail distributions and doesn't hurt performance for datasets with uniform distributions. This could be an interesting method to add to the package.

I would start with running a benchmark on a normal dataset (ImageNet) as we don't have standard long-tail datasets in our benchmarks. It would also be interesting to see if it also gives good performance on full ImageNet as the paper only reports results for ImageNet100-LT. If it also works on default ImageNet it could be a good default method to add to most contrastive models.

TODO:
- [ ] Add optional period to [`cosine_schedule`](https://github.com/lightly-ai/lightly/blob/df004d288fa07816e55975fa0af92868f9867436/lightly/utils/scheduler.py#L7)
- [ ] Add benchmark with SimCLR and temperature schedule
- [ ] Run benchmark and report results
- [ ] Add example and docs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lightly/utils/scheduler.py and the linked paper, then inspect how SimCLR benchmarks are organized in the package. Evaluate the temperature schedule on ImageNet, report the benchmark results, and review the existing example and documentation structure. Done means the optional cosine-schedule period, benchmark, results, example, and docs are all addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.