Feature Requests

@tlienart

1. All Subset (Best K Subset):
   I've got parsimonious code for All Subset Regression which might have a nice home in this package that I'd like to share.
   While it's much faster than R's Leaps pkg, I would include a strong warning to users:
   This optimization problem is sooo not convex. Avoid for p>20.

2. Approximate All Subset (MIO):
   As you probably know,
   Bertsimas etal (2016): show how to use MIO to approximately train all subset regression for p =1000 in minutes.
   Bertsimas et al (2019): do this for p=100k in minutes.
   The corresponding Julia package is SubsetSelection.jl.
   Also see forthcoming Julia package MIP-Boost.
   It would be really awesome if these linear models were featured in MLJ.

3. Relaxed Lasso:
   Hastie et al 2017: responded that relaxed Lasso matches Lasso in low-SNR scenarios & beats All Subset in high-SNR scenarios
   glmnet: now features "relaxed Lasso"
   Would this be difficult to add to MLJLinearModels? (maybe we can work on it together?)

4. Linear Interactions:
   There are some cool new packages for interactions in linear models.
   I find these compare in out-of-sample fit w/ non-linear models (xgboost etc, no joke)
   R: sprintr & HierNet
   Python: HierScale

5. Nonconvex penalties (MCP SCAD):
   https://github.com/joshday/SparseRegression.jl/blob/master/README.md

6. Forward selection/stepwise regression:
   (stackoverflow, has Julia code)
   Textbook code

7. Ensembles of Penalized Linear Models:
   Christidis et al 2019: Paper, old code EnsembleEN, new code SplitReg

The issue is a broad list of seven possible additions, including subset selection, relaxed Lasso, interactions, nonconvex penalties, stepwise regression, and ensembles. Start by choosing one request and reviewing the referenced Julia packages and papers; the work is done only when a single scoped model is integrated into MLJLinearModels with agreed tests and documentation.