NOTE discussion w/ pablin

These are notes from our discussion w/ @pierreablin on the design of the benchmark for NN. Feel free to comment/add/edit stuff.

### Critical steps for CIFAR10 training

There are a few critical steps to watch for good performance when training a neural net on CIFAR:

- Data augmentation is often need. Training with and without it would be a plus.
- The learning rate scheduler can be quite important. Making sure we have the same for the different framework is important.

A source of inspiration for these choice can be: [kuangliu/pytorch-cifar](https://github.com/kuangliu/pytorch-cifar).

### Source of variation between framework

there are a few sources of variation between the framework that will be hard to control:
- Implem of the transforms
- Layer initializations
- Architecture choices (not always the same spot for `BatchNorm`, different drop out, ...)

It is probably fine to not control them completly as this can highlight the differences in some design choices. But it is important to list them well in the paper.

### Implementations to do

- [x] **Comparable loss**: it is fundamental that we make sure the loss is the same for all framework. A way to make sure we completely control this is to wrap all architecture in a class that provides a `predict_proba(X: np.array) -> np.array` function, that return the class probabilities for each samples. That way, we are sure we input the same think and we compute the loss the same.
- [x] **GPU training**: I plan to implement a multi GPU training using `submitit`. I think this should be my priority as this will impact all the chain. I will start from benchopt/benchopt#265 and improve.
- [x] **Multi-framework dataset**: a natural way to support datasets with different framework is to provide for each dataset a implementation on each supported framework. That way, we don't hack our way to convert a dataset loaded in `pytorch` to `tf` and there is no unfair advantage to one framework vs the other. This could for instance be controlled with a parameter `framework` and datasets of improper frameworks can be skipped with [`benchopt.BaseSolver.skip`](https://benchopt.github.io/generated/benchopt.BaseSolver.html#benchopt.BaseSolver.skip). To make the plot possible, see next point.
- [ ] **Multi-parameter plots**: for some parameters in `Objective/Dataset `, one could want to plot all the curves in the same plot. For instance, this is the case with `framework` (see above) or if we use data augmentation or not. Another possiblity woulb be if we have several architectures. Technically, this is something to do at plot time, by simply changing the filtering of what to put on a plot. The big question is on the API side, how to tell which parameters are to be ignored when merging plot together.

### Next step ?

If we have more time for this benchmark, a few ideas that we could try:
- [x] We could do a second benchmark with a single framework but different architectures (resnet18/34/50) to compare them easily.
- [ ] ....

Start with the unchecked Multi-parameter plots item and review the Objective/Dataset plotting discussion, along with the referenced benchopt/benchopt#265 context. Clarify how parameters such as framework or data augmentation should be ignored when merging plots, then verify that combined curves retain the relevant distinctions.

Written by the indexing model from the issue text.