dask / dask/distributed

Poor adaptive target for empty clusters

Open
#6,962 2 comments 0 reactions 0 assignees View on GitHub
adaptive enhancement scheduling
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If a cluster hasn't run any work yet, it will only recommend 1 worker initially, regardless of how many tasks are queued on the scheduler:
```python
@gen_cluster(
client=True,
nthreads=[],
config={"distributed.scheduler.default-task-durations": {"inc": 1}},
)
async def test_adaptive_target_empty_cluster(c, s):
assert s.adaptive_target() == 0

f = c.submit(inc, -1)
await async_wait_for(lambda: s.tasks, timeout=5)
assert s.adaptive_target() == 1

fs = c.map(inc, range(1000))
await async_wait_for(lambda: len(s.tasks) == len(fs) + 1, timeout=5)
print(s.total_occupancy)
> assert s.adaptive_target() > 1
E AssertionError: assert 1 > 1
```

The scheduler's adaptive target is based on looking at its `total_occupancy`. But occupancy is only updated once tasks are scheduled (into processing). So if there are no workers, no tasks can be scheduled, and occupancy remains 0 even with tons of tasks in `unrunnable`.

I would expect the `total_occupancy` to also include the expected runtime of all unrunnable/queued tasks. That would result in faster scale-up from zero usually. Some deployment systems might be quite slow to scale. You might have to wait a few minutes to get 1 worker, to realize you then need more, and then wait a few minutes again. It would be better to ask for more up front.

This is what ensures we at least get one worker, otherwise we'd never scale up at all: https://github.com/dask/distributed/blob/16748b7e32aaa498c3ccb9006ea7237b154d146f/distributed/scheduler.py#L7287-L7288

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.