New Benchmarks

We are in the process of updating our benchmark code and adding benchmarks on the ImageNet dataset. The goal of this issue is to coordinate and track progress of the update.

The new benchmark code lives in lightly/benchmarks and initial focus is on reference implementations for the various SSL methods following the settings from the respective papers. We'll limit training to 100 epochs and small batch sizes (2x128) for now due to limited compute resources.

Note: The benchmarking code is very much work in progress, there will be frequent and breaking changes.

Implemented Methods

• #1452
• W-MSE benchmark
• SSL-EY benchmark PR
• MSN ImageNet Benchmark
• PMSN ImageNet Benchmark
• SimMIM ImageNet Benchmark
• VICRegL ImageNet Benchmark
• NNCLR ImageNet Benchmark
• SMoG ImageNet Benchmark
• SimCLR #1162
• #1230
• #1255
• #1252
• #1287
• #1288
• #1323
• #1303
• #1371
• #1350

Contribute

Contributions are very welcome! You can help us with the following:

• Add one of the existing models to the new benchmarks. For this you can follow the SimCLR example.
• Let us know if you spot any mistake in the models or benchmarking code.
• Run benchmarks for different epochs / batch size.

Let us know if you are working on a model and we can help you out with reviews/compute :)

Start in lightly/benchmarks/imagenet/resnet50 and use the SimCLR benchmark as the reference implementation. Choose one of the unchecked methods, review its corresponding paper settings, and implement or run the ImageNet benchmark with the stated epoch and batch-size limits. Done means the method is benchmarked and its results or implementation are ready for review.