Deep Double Descent: Where Bigger Models and More Data Hurt

https://arxiv.org/pdf/1912.02292.pdf

Double descent has been previously been discussed in terms of the number of parameters of a neural network. While in classical statistics world you eventually start overfitting the data if you have too many parameters, in ML you can power through and get monotonically increasing performance as long as you have a lot of parameters.

Nakkiran et al. found that this behavior isn't just affected by the number of parameters, but also by the amount of data in the training set, the number of epochs trained, the amount of noise in the dataset, etc.

Interesting takeaways:
- If your model works well because it's overparametrized, adding data can actually decrease your performance as you'll need even more parameters to return to the same performance
- If you use early stopping you don't really have to worry about any of the things in this paper, but it's unclear whether you could do better than early stopping by training for a very long time
- The authors argue that because of the many factors involved in double descent/overfitting, model complexity should really be thought of in terms of the number of training set datapoints the model can fit perfectly