YarnCluster() does not initialize but runs indefinetly
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
dask-yarn is not able to initialize the cluster, it runs indefinetly
**What you expected to happen**:
Cluster being initialized correctly
**Minimal Complete Verifiable Example**:
```python
from dask_yarn import YarnCluster
from dask.distributed import Client
cluster = YarnCluster(environment='path/to/env.tar.gz',
worker_vcores=2,
worker_memory="8GiB")
cluster.scale(2)
client = Client(cluster)
```
**Anything else we need to know?**:
Skein runs and is able to submit YARN jobs successfully.
Console Log:
```
/home/jovyan/.local/lib/python3.7/site-packages/dask_yarn/core.py:16: FutureWarning: format_bytes is deprecated and will be removed in a future release. Please use dask.utils.format_bytes instead.
from distributed.utils import (
/home/jovyan/.local/lib/python3.7/site-packages/dask_yarn/core.py:16: FutureWarning: parse_timedelta is deprecated and will be removed in a future release. Please use dask.utils.parse_timedelta instead.
from distributed.utils import (
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
22/06/15 12:13:51 INFO client.RMProxy: Connecting to ResourceManager at hostname.local/xx.xxx.xx.x:8032
22/06/15 12:13:52 INFO skein.Driver: Driver started, listening on 42731
22/06/15 12:13:52 INFO conf.Configuration: resource-types.xml not found
22/06/15 12:13:52 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/06/15 12:13:52 INFO skein.Driver: Uploading application resources to hdfs://user/jovyan/.skein/application_1654689977550_0381
22/06/15 12:13:52 INFO skein.Driver: Submitting application...
22/06/15 12:13:52 INFO impl.YarnClientImpl: Submitted application application_1654689977550_0381
```
YARN shows this job as finished.
Using a ``python://path/to/binary`` or ``hdfs://path/to/tar.gz`` or ``path/to/tar.gz`` has no influence on this issue.
**Environment**:
- Dask version: 0.9
- Python version: 3.7.12
- Operating System: Linux 5.4.17-2136.307.3.6.el8uek.x86_6
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.