dask / dask/dask-ml

DaskML vs. Sklearn LogisticRegression. Coefficients seem different.

Open
#860 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

The coefficients from the `daskml.linear_model.LogisticRegression` seem different from the `sklearn.linear_model.LogisticRegression`.

The script below demonstrates a benchmark that shows this.

```python
# %pip install dask_ml memo
import time
import numpy as np
from dask_ml.datasets import make_classification
from dask_glm.linear_model import LogisticRegression as DaskLogReg
from sklearn.linear_model import LogisticRegression as SklearnLogReg
from memo import grid, memlist, memfunc

data_hindsight = []

@memfunc(print)
@memlist(data=data_hindsight)
def run_experiment(size=1_000, chunksize=1_000):
X, y = make_classification(n_samples=size, n_features=20, n_informative=10, chunksize=chunksize)

# Run everything in Dask
tic = time.time()
mod_dask = DaskLogReg()
mod_dask.fit(X, y)
dask_time = time.time() - tic

# Run everything in Scikit-Learn
X_np, y_np = np.asarray(X), np.asarray(y)
mod_sklearn = SklearnLogReg()
tic = time.time()
mod_sklearn.fit(X_np, y_np)
sklearn_time = time.time() - tic

# Return stats that are of interest
return {
'time_dask': dask_time,
'time_sklearn': sklearn_time,
'acc_dask': (mod_dask.predict(X) == y).mean().compute(),
'acc_sklearn': (mod_sklearn.predict(X_np) == y_np).mean(),
'coef_diff': np.abs(mod_sklearn.coef_ - mod_dask.coef_).sum() # This is what I'm interested in understanding.
}

for size in [1000, 2000]:
for chunksize in [500, 1000]:
run_experiment(size=size, chunksize=chunksize)
```

The output in `data_hindsight` looks like this:

```
{'size': 1000, 'chunksize': 500, 'time_dask': 1.2568867206573486, 'time_sklearn': 0.003863811492919922, 'acc_dask': 0.795, 'acc_sklearn': 0.795, 'coef_diff': 7.47177509238394}
{'size': 1000, 'chunksize': 1000, 'time_dask': 0.09700465202331543, 'time_sklearn': 0.005501270294189453, 'acc_dask': 0.784, 'acc_sklearn': 0.784, 'coef_diff': 8.645457699298978}
{'size': 2000, 'chunksize': 500, 'time_dask': 2.3598458766937256, 'time_sklearn': 0.007267951965332031, 'acc_dask': 0.757, 'acc_sklearn': 0.7605, 'coef_diff': 5.962742662561093}
{'size': 2000, 'chunksize': 1000, 'time_dask': 1.5501070022583008, 'time_sklearn': 0.010750293731689453, 'acc_dask': 0.7985, 'acc_sklearn': 0.7995, 'coef_diff': 8.450044499024445}
```

Although the accuracy on the train set seems to be roughly the same, I cannot help but wonder why the `coef_diff` is so large. I understand that the internal optimizer is bound to be different in a parallel setting so there's gotta some difference here, but the difference seem too big for my gut feeling.

**Environment**:

```
Python implementation: CPython
Python version : 3.7.9
IPython version : 7.27.0

Compiler : GCC 9.3.0
OS : Linux
Release : 5.11.0-7614-generic
Machine : x86_64
Processor : x86_64
CPU cores : 12
Architecture: 64bit

dask==2021.9.1
dask-glm==0.2.0
dask-ml==1.9.0
scikit-learn==0.24.2
```

Contributor guide

Open the contributing guide

Research direction

The reproduction is in run_experiment and compares dask_ml.linear_model.LogisticRegression with sklearn.linear_model.LogisticRegression. Start by running it with the listed package versions, then inspect the two estimators' fit inputs and documented defaults. Done means the coefficient discrepancy is explained and the expected behavior is recorded or corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.