Excessive memory usage in logistic regression
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
```python
from dask_yarn import YarnCluster
from dask.distributed import Client
# Create a cluster where each worker has two cores and eight GiB of memory
cluster = YarnCluster(environment='s3://openbank-ds-playground/environments/conda/gru13-07.tar.gz',
worker_vcores=1,
worker_memory="50GiB",
deploy_mode='local',
dashboard_address=':6689',
)
cluster.adapt(minimum=4, maximum=10)
client = Client(cluster)
```
**Hi guys, I've notice a weird behaviour in my dask application. I'm running a Logistic Regression with dask-ml in the YarnCluster I have created above over EMR architecture, and I can see that each worker takes like 15 times the memory of the dataset and i'm specifying to work only with 1 vcore in each worker. I've tested with different dataset sizes and always get into 10-20 times the size of the dataset. The data is loaded from S3 through pandas and s3fs. I can't figure out why is this happening. Could you help me?**:
**Environment**:
- Dask version: 2.18.0
- dask_yarn version: 0.8.1
- Python version: 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21)
[GCC 7.3.0]
Contributor guide
Assessment
This issue has not been assessed yet.