dask / dask/dask-ml

Sklearn random forest does not train on dask yarn cluster

Open
#753 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
I am attempting to train a model using dask-ml. I've had success using ParallelPostFit, ColumnTransformer, and related dask-ml pipeline utilities to train and predict using sklearn's LogisticRegression as the classifier. However, when I've tried to use sklearn's RandomForestClassifier I have run into issues. Here I have tried to create a minimal example. The code below runs successfully when the classifier is LogisticRegression, but hangs when the classifier is RandomForest. I don't know why this is happening. I am running on a dask yarn cluster. Thank you very much in advance for your help!

**Minimal Complete Verifiable Example**:

```
import dask
import dask_yarn
import dask.array as da
from dask.distributed import LocalCluster, Client, progress
from dask_yarn import YarnCluster
from dask_ml.datasets import make_classification as make_classification_dask
from dask_ml.model_selection import train_test_split as train_test_split_dd

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
import joblib

worker_env={
'ARROW_LIBHDFS_DIR': '/usr/hdp/2.6.1.40-4/usr/lib',
'HADOOP_HOME': '/usr/hdp/current/hadoop-client'
}
cluster = YarnCluster(environment=path_to_packed_conda_env,
n_workers=10,
worker_vcores=10,
worker_env=worker_env,
worker_restarts=10,
scheduler_memory='10GiB',
scheduler_vcores=5,
worker_memory='10GiB')

client = Client(cluster)

RANDOM_STATE = 42

clf_rfc = RandomForestClassifier(n_estimators=10,
n_jobs=5,
criterion='gini',
max_features='auto',
min_samples_split = 50,
class_weight='balanced',
verbose=1,
random_state=RANDOM_STATE)

clf = LogisticRegression(penalty='l2',
dual=False,
tol=0.0001,
C=0.1,
fit_intercept=True,
intercept_scaling=1,
#class_weight='balanced',
random_state=RANDOM_STATE,
solver='liblinear',
max_iter=100,
multi_class='auto',
verbose=0,
warm_start=False,
n_jobs=-1)

# make 4 Gb dataset
n, d = 10000000, 50

X, y = make_classification_dask(n_samples=n,
n_features=d,
chunks=n//10,
flip_y=0.2,
random_state=0)

X_train, X_test, y_train, y_test = train_test_split_dd(X, y, train_size = 0.8, random_state=RANDOM_STATE)

# pipe = clf
pipe = clf_rfc

X_train = X_train.persist()
y_train = y_train.persist()

with joblib.parallel_backend('dask'):
pipe.fit(X_train, y_train)
```

**Environment**:

- Dask version: 2.11
- Dask ML version: 1.4.0
- Python version: 3.7
- Operating System: Linux
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal example with RandomForestClassifier and LogisticRegression on the Dask Yarn cluster, comparing the behavior around joblib.parallel_backend('dask') and pipe.fit(X_train, y_train). Use the reported Dask, Dask-ML, Python, and Linux versions; done means the random forest training completes instead of hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.