LinearRegression returns error for 1e7 elements
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
dask_ml.linear_model.LinearRegression doesn't work for 1e7 elements (while it works for size = 1e6):
```
from sklearn.pipeline import make_pipeline
from dask_ml.preprocessing import StandardScaler
from dask_ml.linear_model import LinearRegression
size = 1e7
X = dask.array.arange(2*size).reshape(-1,2)
y = dask.array.arange(size).reshape(-1,1)
reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X, y)
```

sklearn.linear_model.LinearRegression works well for the same case:
```
from sklearn.pipeline import make_pipeline
#from dask_ml.preprocessing import StandardScaler
#from dask_ml.linear_model import LinearRegression
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
size = 1e7
X = dask.array.arange(2*size).reshape(-1,2)
y = dask.array.arange(size).reshape(-1,1)
reg = make_pipeline(StandardScaler(), LinearRegression())
reg.fit(X, y)
```

Contributor guide
Research direction
Start by running the provided dask-ml pipeline with size = 1e7 and compare it with the sklearn pipeline. Inspect the dask_ml.preprocessing.StandardScaler and dask_ml.linear_model.LinearRegression entry points to identify why fitting fails at that size; done means the dask-ml pipeline fits successfully for 1e7 elements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100