dask / dask/distributed

Semantic mismatch in `SpecCluster.requested`

Open
#9,103 2 comments 0 reactions 0 assignees View on GitHub
needs info
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

related to #9102

`SpecCluster.requested` doesn't match what `AdaptiveCore` expects:
- AdaptiveCore expects: "workers we've asked for but haven't arrived yet"
- SpecCluster provides: "all workers in our spec, expanded by groups"

This mismatch exists because SpecCluster uses self.workers as a proxy for "requested". For non-grouped workers these are 1:1. For grouped workers, a single worker in self.workers is a group of multiple worker processes.

## potential fix

We could make `SpecCluster.requested` more accurately represent "workers we've asked for that the scheduler knows about"

```python
@property
def requested(self):
out = set()
scheduler_workers = {d["name"] for d in self.scheduler_info.get("workers", {}).values()}

for name in self.workers:
try:
spec = self.worker_spec[name]
except KeyError:
continue

if "group" in spec:
# Only count workers that actually exist
out.update({
str(name) + suffix
for suffix in spec["group"]
if str(name) + suffix in scheduler_workers
})
else:
if name in scheduler_workers:
out.add(name)
return out
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.