dask / dask/distributed

Creating LocalCluster with processes through reticulate in R hangs on Windows

Open
#5,461 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The following R program hangs on Windows:

```R
library(reticulate)
d <- import("dask.distributed")
lc <- d$LocalCluster(threads_per_worker=1L, processes=T)
```

**What works**:

* changing to `processes=F`
* running the same example in python in the same reticulate virtual environment
* Running the same in R on Linux

**Faultfinding**:

I could trace the hang to the `_correct_state_internal` method in the `deploy/sync.py` module. See the marked line near the end below. I would be happy to attempt more fault finding, but I would need help with ideas on what to try.

```python
async def _correct_state_internal(self):
async with self._lock:
self._correct_state_waiting = None

to_close = set(self.workers) - set(self.worker_spec)
if to_close:
if self.scheduler.status == Status.running:
await self.scheduler_comm.retire_workers(workers=list(to_close))
tasks = [
asyncio.create_task(self.workers[w].close())
for w in to_close
if w in self.workers
]
await asyncio.wait(tasks)
for task in tasks: # for tornado gen.coroutine support
with suppress(RuntimeError):
await task
for name in to_close:
if name in self.workers:
del self.workers[name]

to_open = set(self.worker_spec) - set(self.workers)
workers = []
for name in to_open:
d = self.worker_spec[name]
cls, opts = d["cls"], d.get("options", {})
if "name" not in opts:
opts = opts.copy()
opts["name"] = name
if isinstance(cls, str):
cls = import_term(cls)
worker = cls(self.scheduler.address, **opts)
self._created.add(worker)
workers.append(worker)
if workers:
###### The following line hangs #########
await asyncio.wait(workers)
for w in workers:
w._cluster = weakref.ref(self)
await w # for tornado gen.coroutine support
self.workers.update(dict(zip(to_open, workers)))
```

**Environment**:

- Dask version: 2021.10.0
- Python version: 3.9.5
- Operating System: Windows 10
- Install method (conda, pip, source): pip (into reticulate conda environment)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.