sklearn StandardScaler vs dask StandardScaler.
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
I am getting different results from sklearn StandardScaler and dask StandardScaler.
```
scaler_sk = sklearn.preprocessing.StandardScaler()
scaler_d = dask_ml.preprocessing.StandardScaler()
scaler_sk.fit(df_pd[["SUMMESSAGECOUNT"]])
scaler_d.fit(df_dask[["SUMMESSAGECOUNT"]])
```
Dask scaler
```
scaler_d.mean_[0], scaler_d.var_[0]
output: (19.157653421114507, 47431.17794342375)
```
Sklearn Scaler
```
scaler_sk.mean_[0], scaler_sk.var_[0]
output: (19.157653421114507, 47431.17794342373)
```
I know the difference is negligible. But it is influencing my model training on prophet. Could you please suggest any way to make them identical without using `compute()`.
Contributor guide
Research direction
Start by reproducing the reported fit using sklearn.preprocessing.StandardScaler and dask_ml.preprocessing.StandardScaler on the same single-column data, without calling compute(). Compare their mean_ and var_ values and trace where each implementation performs its calculation. Done means establishing whether the discrepancy is expected and identifying a documented or actionable way to obtain matching results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100