alteryx / alteryx/evalml

ClassImbalanceDataCheck should use n_folds instead of 2 * n_folds

Open
#2,995 0 comments 0 reactions 1 assignee Claimed by @asniyaz View on GitHub
bug
Dominant language
Python
Stars
850
Forks
96
PR merge metrics
No merged PRs in 30d

Description

Repro with the `abalone_trimmed.csv` dataset. I manually modified it so that the least common class has 9 instances.

```python
from evalml.automl.utils import make_data_splitter
from evalml.automl import AutoMLSearch
from evalml.data_checks import ClassImbalanceDataCheck
import pandas as pd
import pytest

X = pd.read_csv("/Users/freddy.boulton/Downloads/abalone_trimmed.csv")
y = X.pop(" rings")

assert y.value_counts().min() == 9

assert ClassImbalanceDataCheck(num_cv_folds=5).validate(X, y)['errors'][0]["code"] == 'CLASS_IMBALANCE_BELOW_FOLDS'

splitter = make_data_splitter(X, y, problem_type="multiclass", n_splits=5)

automl = AutoMLSearch(X, y, "multiclass", data_splitter=splitter, max_iterations=3)
automl.search()

assert ClassImbalanceDataCheck(num_cv_folds=10).validate(X, y)['errors'][0]["code"] == 'CLASS_IMBALANCE_BELOW_FOLDS'

splitter = make_data_splitter(X, y, problem_type="multiclass", n_splits=10)

automl = AutoMLSearch(X, y, "multiclass", data_splitter=splitter, max_iterations=3)

with pytest.raises(Exception, match="Missing target values"):
automl.search()
```

Note that using 5 folds does not cause an error even though we raise a data check error. Using n_splits=10 does cause an error. I think we should use `n_folds` instead of `2 * n_folds` in the `ClassImbalanceDataCheck`, sklearn also warns us about being less than `n_folds` not `2 * n_folds`

![image](https://user-images.githubusercontent.com/41651716/139502873-3bf79f8d-4c3d-4cbd-9f59-460a55351d07.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.