ContinualAI / ContinualAI/avalanche

feature request: semi-supervised, unsupervised, and novelty detection common benchmarks

Open
#1,329 1 comment 0 reactions 0 assignees View on GitHub
Benchmarks Feature - Medium Priority Training
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

More and more CL papers are tackling unsupervised and a mix of supervised and unsupervised learning in the different tasks. I don't think there is a clean way to address that in avalanche right now so I'm proposing an idea.

I think it could be a nice idea to propose a general benchmark-maker, or an option to existing generators, where you decide how many labels in % are available for each task, with the option to chose which one of these labels are available, or to just set them at random.

As an example if we have five tasks here is how it would go for different existing problems

Setting | Label % per task (i.e 3 tasks)

Unsupervised | 0 | 0 | 0
Semi-Supervised | 5% | 5% | 5%
New Class Discovery (NCD) | 100% (pretraining task) | 0% | 0%
A mix of NCD and Semi-Supervised | 100% | 5% | 5%

That way it should be pretty easy to implement all of these settings that play with the availability of the labels. This could be implemented by a simple masking of which samples labels should be replaced by a given value (-1?) and then strategies could filter out what part of the input they can train on based on the labels in the current experience, for instance, a SupervisedTemplate would only treat the samples that have label, by means of self.mbatch = self.mbatch[self.mbatch[1] != -1] or something like that.

This definitely needs more thinking but I think a systematic treatment of these kind of scenarios could be interesting to integrate

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.