dask / dask/dask-glm

Profile results

Open
#26 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
78
Forks
47
PR merge metrics
No merged PRs in 30d

Description

On eight m4.2xlarges I created the following dataset

```python
N = 1e8
beta = np.array([-1, 0, 1, 2])
M = 4
chunks = 1e6
seed = 20009

X = da.random.random((N, M), chunks=(chunks, M))
z0 = X.dot(beta)
y = da.random.random(z0.shape, chunks=z0.chunks) < sigmoid(z0)

X, y = persist(X, y)
```

I then ran the various methods within this project and recorded the profiles as bokeh plots. They are linked to below:

- [admm](https://cdn.rawgit.com/mrocklin/e7bcb979e147102bf9ac428ed9074000/raw/d38234f3e9072816bea98d032f1e4f9e618242c3/task-stream-glm-admm.html)
- [gradient-descent](https://cdn.rawgit.com/mrocklin/e7bcb979e147102bf9ac428ed9074000/raw/d38234f3e9072816bea98d032f1e4f9e618242c3/task-stream-glm-gradient-descent.html)
- [newton](https://cdn.rawgit.com/mrocklin/e7bcb979e147102bf9ac428ed9074000/raw/d38234f3e9072816bea98d032f1e4f9e618242c3/task-stream-glm-newton.html)
- [bfgs](https://cdn.rawgit.com/mrocklin/e7bcb979e147102bf9ac428ed9074000/raw/d38234f3e9072816bea98d032f1e4f9e618242c3/task-stream-glm-bfgs.html)
- [proximal-descent](https://cdn.rawgit.com/mrocklin/e7bcb979e147102bf9ac428ed9074000/raw/d38234f3e9072816bea98d032f1e4f9e618242c3/task-stream-glm-proximal-grad.html)

Additionally, I ran against a 10x larger dataset and got the following results

- [proximal grad](https://cdn.rawgit.com/mrocklin/77fec28c36238f81c7bccde6af3b09dd/raw/3f0dce8ee240434e73864ec6d341b9270c244f2d/task-stream-glm-proximal-grad-large.html)

Most runtimes were around a minute. The BFGS solution gave wrong results.

### Notes

On larger problems with smallish chunks (8 * 4 * 1e6 == 24 MB) we seem to be bound by scheduling overhead. I've created an isolated benchmark here that is representative of this overhead: https://gist.github.com/mrocklin/48b7c4b610db63b2ee816bd387b5a328

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.