More Robust Support for LeaveOneOut (with ROC AUC, FBeta, recall, precision)

This is not an issue with TPOT per se, but I would like to use TPOT with LeaveOneOut validation and evaluate the scores using ROC AUC.

Technically, this is a limitation of sklearn's cross_val_score, which TPOT uses. When using LeaveOneOut, sklearn's cross_val_score errors when the "scoring" parameter is set to 'roc_auc' (as it should, as in each instance there is only one class present, and roc_auc cannot be calculated for a single sample). Naturally, this means that when using TPOTClassifier and setting the "cv" parameter to LeaveOneOut() and the "scoring" parameter to 'roc_auc' will give the same error. (For what it's worth, some more informative error messages here would be a significant help in diagnosing what went wrong in the scoring process, instead of the relatively uninformative message below).


To be more specific, cross_val_score can only be safely used within TPOTClassifier with LeaveOneOut when the "scoring" parameter is set to a metric that can be evaluated on a single sample, all predictions for all samples pooled, and then the mean of those predictions taken. This is true for accuracy, but **not** for recall, precision, fbeta (which do not result in an error) or roc_auc (which does result in an error). I suspect there are similar instances with regression metrics also (perhaps r2), though I have not looked into this.

A fix would involve pooling the predictions for all samples together prior to evaluating the given metric when the metric cannot be accurately evaluated by pooling single sample metrics and taking the mean. I believe this would mean a move away from cross_val_score in those cases.

Start by reproducing TPOTClassifier with LeaveOneOut() and scoring='roc_auc', then inspect where TPOT uses sklearn's cross_val_score and how the scoring metrics behave. Done means LeaveOneOut can evaluate ROC AUC, recall, precision, and F-beta using pooled predictions, with more informative errors for unsupported scoring cases.

Written by the indexing model from the issue text.