dask / dask/dask-ml

LinearRegression returns error for 1e7 elements

Open
#948 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

dask_ml.linear_model.LinearRegression doesn't work for 1e7 elements (while it works for size = 1e6):
```
from sklearn.pipeline import make_pipeline
from dask_ml.preprocessing import StandardScaler
from dask_ml.linear_model import LinearRegression

size = 1e7
X = dask.array.arange(2*size).reshape(-1,2)
y = dask.array.arange(size).reshape(-1,1)
reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X, y)
```
image

sklearn.linear_model.LinearRegression works well for the same case:

```
from sklearn.pipeline import make_pipeline
#from dask_ml.preprocessing import StandardScaler
#from dask_ml.linear_model import LinearRegression
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler

size = 1e7
X = dask.array.arange(2*size).reshape(-1,2)
y = dask.array.arange(size).reshape(-1,1)
reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X, y)
```
image

Contributor guide

Open the contributing guide

Research direction

Start by running the provided dask-ml pipeline with size = 1e7 and compare it with the sklearn pipeline. Inspect the dask_ml.preprocessing.StandardScaler and dask_ml.linear_model.LinearRegression entry points to identify why fitting fails at that size; done means the dask-ml pipeline fits successfully for 1e7 elements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.