Tips and Tricks for training classification convolutional neural networks

Data augmentation:

- [ ] Random Resized Crop
- [x] Horizontal Flip
- [ ] Random Augment (see [code](https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py#L672), [paper](https://arxiv.org/abs/1909.13719) ), with e.g. 7/0.5. Basically it is a fixed set of data augmentation functions.
- [ ] Color Jitter
- [ ] PCA lighting
- [ ] Random Erasing
- [ ] Mixup
- [ ] Cutmix

Regularization:

- [ ] Label smoothing (basically making the target of classif not (1, 0, 0 ,0) but (1-e, e/3, e/3, e/3)
- [ ] Repeated Augmentation (multiple instances of a sample in the same batch, with different augmentation)
- [ ] stochastic depth (basically drop some blocks and replace them with identity stochastically). It sounds very complicated to use as part of a solver in our setup. Moreover, the "ResNet strikes back" paper claims that this is only a good idea for very big networks, so it might not be our priority.
- [x] weight decay (according to "Bag of tricks" this might only be useful for non-BN params). I have seen somewhere (but don't remember where exactly) that it should also only be applied to the weights and not the biases.

Learning rate:

- [ ] Warmup: scale linearly the learning rate from 0 to the initial in the first few (5) epochs.
- [x] Scheduling: step or cosine. The problem with the cosine schedule is that it needs to know the total number of epochs, which in the current setting is not available to the solver. Could we work around that @tomMoral ?

Modeling (to me these ones are out of our scope):

- [ ] Zero gamma: make the initialization of the learned scale of BN layers to 0.
- [ ] Layer Scale (basically learn a multiplicative factor per channel at the end of residual blocks)

Other:

- [ ] mixed precision
- [ ] weight averaging (SWA or EMA)
- [ ] Binary Xent rather than categorical Xent, coupled with Cutmix and Mixup, in a 1-vs-all fashion
- [ ] gradient clipping (not used in "ResNet strikes back")

No file, test, or entry point is named. Start by inspecting the repository's existing training setup and comparing it with the unchecked augmentation, regularization, learning-rate, and other ideas listed here. Done would require narrowing the checklist to a defined feature, implementing it in the benchmark, and validating its effect.

Written by the indexing model from the issue text.