Regularization in TPOT

Some months ago, we added Pareto optimization to TPOT, where TPOT now maximizes the pipeline classification accuracy while minimizing the number of operators in the pipeline. We found that such an addition provided a form of regularization for TPOT: the pipelines that TPOT produced were less likely to overfit on the data set.

As I've read more about regularization, I'm starting to wonder if we should refine what we mean by "pipeline complexity" in TPOT. Although "number of operators in the pipeline" is a decent metric for pipeline complexity, maybe we should consider more traditional measures of model complexity.

The first idea that comes to mind is the **number of features going into the final classifier**. Such a regularization metric could encourage TPOT to compress the feature space (e.g. via PCA or feature construction), perform feature selection in the second-to-last step, and thus build less-complex models that are less prone to overfitting.

Please add additional TPOT regularization ideas to this issue.