dask / dask/dask-ml

sklearn StandardScaler vs dask StandardScaler.

Open
#979 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

I am getting different results from sklearn StandardScaler and dask StandardScaler.
```
scaler_sk = sklearn.preprocessing.StandardScaler()
scaler_d = dask_ml.preprocessing.StandardScaler()

scaler_sk.fit(df_pd[["SUMMESSAGECOUNT"]])
scaler_d.fit(df_dask[["SUMMESSAGECOUNT"]])
```

Dask scaler
```
scaler_d.mean_[0], scaler_d.var_[0]
output: (19.157653421114507, 47431.17794342375)
```
Sklearn Scaler
```
scaler_sk.mean_[0], scaler_sk.var_[0]
output: (19.157653421114507, 47431.17794342373)
```
I know the difference is negligible. But it is influencing my model training on prophet. Could you please suggest any way to make them identical without using `compute()`.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported fit using sklearn.preprocessing.StandardScaler and dask_ml.preprocessing.StandardScaler on the same single-column data, without calling compute(). Compare their mean_ and var_ values and trace where each implementation performs its calculation. Done means establishing whether the discrepancy is expected and identifying a documented or actionable way to obtain matching results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.