dask / dask/dask-ml

Unpredictable results when some NaNs included in input

Open
#947 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

NaN values can't be used to fit a model and usually should be excluded. But dask_ml allows NaNs just to return wrong output:

```
from sklearn.pipeline import make_pipeline
from dask_ml.preprocessing import StandardScaler
from dask_ml.linear_model import LinearRegression

X = 1.*np.array([[1, 1], [1, 2], [2, 1], [2, 2]])
y = np.array([ 6., 8., 9., np.nan])

reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X, y)
print (reg.predict(np.array([[3., 5.]])))
[10.9140625]

reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X[~dask.array.isnan(y)], y[~dask.array.isnan(y)])
print (reg.predict(np.array([[3., 5.]])))
[15.54899511]
```

Sure, `reg.fit(X[~dask.array.isnan(y)], y[~dask.array.isnan(y)])` is the correct way but it works extremely slow on big datasets due to slow y[~dask.array.isnan(y)] or `y[~np.isnan(y)]` calculation. It'd be nice to allow dask_ml ignore NaNs but the result is wrong. While dask_ml provides `SimpleImputer(strategy='mean')` that's a terrible idea to use 1D mean value to fill multidimensional data gaps.

Maybe is there any approach to just exclude NaNs for dask_ml functions by scalable way?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.