dask / dask/dask-ml

For a single record data frame train_test_split() sometimes assigns this single record to test set.

Open
#975 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

_Disclaimer: I know the bug looks silly but I still wanted to give a heads up._

For a single data frame with only 1 record `train_test_split()` sometimes returns empty train set and test set with 1 record - is that desired behavior?

**Minimal Complete Verifiable Example**:

```python
import pandas as pd
import dask.dataframe as dd
from dask_ml.model_selection import train_test_split

if __name__ == '__main__':

for _ in range(20):

df = pd.DataFrame({'x0': [0], 'x1': [1], 'y': [2]})

ddf = dd.from_pandas(df, npartitions=1)
x = ddf[['x0', 'x1']]
y = ddf['y']

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3)

if x_train.shape[0].compute() == 0:
print('x_train is empty!')
break

```

**Anything else we need to know?**:

Nope

**Environment**:

- Dask version: 2023.5.0
- Dask ML version: 2023.3.24
- Python version: 3.8.15
- Operating System: Ubuntu 22.04
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Research direction

Start by running the issue's minimal example with dask.dataframe and dask_ml.model_selection.train_test_split, checking how often the one-record input produces an empty training set. Determine the intended behavior for a one-record split, then add or update coverage around that case and verify the split results consistently match the decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.