dask / dask/dask-ml

Excessive memory usage in logistic regression

Open
#717 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

```python
from dask_yarn import YarnCluster
from dask.distributed import Client

# Create a cluster where each worker has two cores and eight GiB of memory
cluster = YarnCluster(environment='s3://openbank-ds-playground/environments/conda/gru13-07.tar.gz',
worker_vcores=1,
worker_memory="50GiB",
deploy_mode='local',
dashboard_address=':6689',
)

cluster.adapt(minimum=4, maximum=10)
client = Client(cluster)
```

**Hi guys, I've notice a weird behaviour in my dask application. I'm running a Logistic Regression with dask-ml in the YarnCluster I have created above over EMR architecture, and I can see that each worker takes like 15 times the memory of the dataset and i'm specifying to work only with 1 vcore in each worker. I've tested with different dataset sizes and always get into 10-20 times the size of the dataset. The data is loaded from S3 through pandas and s3fs. I can't figure out why is this happening. Could you help me?**:

**Environment**:

- Dask version: 2.18.0
- dask_yarn version: 0.8.1
- Python version: 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21)
[GCC 7.3.0]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.