alteryx / alteryx/evalml

ClassImbalanceDataCheck should use n_folds instead of 2 * n_folds

Ouverte
#2,995 0 commentaires 0 réactions 1 personne assignée Réclamée par @asniyaz Voir sur GitHub
bug
Langage dominant
Python
Étoiles
850
Forks
96
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Repro with the `abalone_trimmed.csv` dataset. I manually modified it so that the least common class has 9 instances.

```python
from evalml.automl.utils import make_data_splitter
from evalml.automl import AutoMLSearch
from evalml.data_checks import ClassImbalanceDataCheck
import pandas as pd
import pytest

X = pd.read_csv("/Users/freddy.boulton/Downloads/abalone_trimmed.csv")
y = X.pop(" rings")

assert y.value_counts().min() == 9

assert ClassImbalanceDataCheck(num_cv_folds=5).validate(X, y)['errors'][0]["code"] == 'CLASS_IMBALANCE_BELOW_FOLDS'

splitter = make_data_splitter(X, y, problem_type="multiclass", n_splits=5)

automl = AutoMLSearch(X, y, "multiclass", data_splitter=splitter, max_iterations=3)
automl.search()

assert ClassImbalanceDataCheck(num_cv_folds=10).validate(X, y)['errors'][0]["code"] == 'CLASS_IMBALANCE_BELOW_FOLDS'

splitter = make_data_splitter(X, y, problem_type="multiclass", n_splits=10)

automl = AutoMLSearch(X, y, "multiclass", data_splitter=splitter, max_iterations=3)

with pytest.raises(Exception, match="Missing target values"):
automl.search()
```

Note that using 5 folds does not cause an error even though we raise a data check error. Using n_splits=10 does cause an error. I think we should use `n_folds` instead of `2 * n_folds` in the `ClassImbalanceDataCheck`, sklearn also warns us about being less than `n_folds` not `2 * n_folds`

![image](https://user-images.githubusercontent.com/41651716/139502873-3bf79f8d-4c3d-4cbd-9f59-460a55351d07.png)

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.