Train Test Split Unexpected Behavior
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
`_validate_shuffle_split` is not setting the size of test correctly when only `train_size` is included. The reverse is also true, train size will be wrong if you only set `test_size`.
I passed a dask array to `train_test_split` from `dask-ml`. The `_validate_shuffle_split` method in this function is actually imported from sklearn. (It's possible that this may mean the issue is better posed to sklearn but as it's affecting dask users you might be interested.)
The error you should see when running the reprex is: `"ValueError: With n_samples=1, test_size=0.30000000000000004 and train_size=0.7, the resulting train set will be empty. Adjust any of the aforementioned parameters."` This error would be entirely accurate if it said `test_size = 0.3` but it's clearly not right as it is. If you manually set `train_size` and `test_size` both, then the error is correct and behavior is fine.
The smallest specified value that shows an issue as far as I can tell is .66.
Given the incredibly small decimal error on the split default, if you run correct data the error doesn't show itself. However, at scale it eventually results in the test set being microscopically larger than it should be, so this error surfaced in my dask use case because the data was large enough for the test size to calculate out to one extra observation.
**What you expected to happen**:
dask-ml version of `train_test_split` would default the size of test to 0.3 exactly, not 0.30000000000000004.
**Minimal Complete Verifiable Example**:
```python
from dask_ml.model_selection import train_test_split
import dask.dataframe as dd
from sklearn import datasets
import pandas as pd
import numpy as np
# Grab sample dataset
iris = datasets.load_iris()
iris_pd = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
columns= iris['feature_names'] + ['target'])
iris_pd['col1_madeup'] = iris_pd['sepal length (cm)'].astype(str)
iris_pd['col2_madeup'] = iris_pd['sepal width (cm)'].astype(str)
iris_dd = dd.from_pandas(iris_pd, npartitions=10)
def preprocess(dns: dd.DataFrame) -> dd.DataFrame:
dns['target'] = 0
dns = dns.groupby(["target", "col2_madeup"]).col1_madeup.apply(list)
dns = dns.reset_index()
return dns
iris_dd = preprocess(iris_dd)
iris_da = iris_dd.to_dask_array().compute_chunk_sizes()
x_train, x_test, y_train, y_test = train_test_split(
iris_da[:,1],
iris_da[:,2],
train_size=0.7,
#test_size = 0.3, # If you comment this out, the sizes are corrected.
shuffle=True,
)
```
**Anything else we need to know?**:
~It seems like the solution choices are to write your own version of `_validate_shuffle_split` or I can go over to sklearn and investigate with them, to try and get a fix introduced to their version that dask-ml can just import.~
Never mind, sounds like this a floating point arithmetic dilemma more likely. So some arithmetic remedy like `float()` might be the right choice.
**Environment**: Reproduced on Macbook Pro locally and also on Saturn Cloud Jupyter Labs notebook
- Dask version: 2.27.0
- Python version: 3.8
- Operating System: Mac OS 10.15.6
- Install method (conda, pip, source): pip
Contributor guide
Research direction
Start at dask-ml's train_test_split entry point and inspect the imported sklearn _validate_shuffle_split helper. Run the minimal example from the issue with only train_size set, then compare the computed sizes with the case where both sizes are supplied. Done means the default test size is treated as exactly 0.3 and the resulting split sizes are correct at scale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100