dask / dask/distributed

Adaptive target duration is scaled by `threads_per` worker

Open
#2,709 2 comments 0 reactions 0 assignees View on GitHub
adaptive
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Adaptive Clusters with `threads_per_worker > 1` will scale to reach a duration `target_duration / threads_per_worker`:

```python
from dask.distributed import LocalCluster, Client, Adaptive
from dask import bag as db
from time import time, sleep

cluster = LocalCluster(n_workers=1, threads_per_worker=2, memory_limit=1e9)
ca = cluster.adapt(minimum=1, maximum=10, target_duration="20s", scale_factor=1)

client = Client(cluster)

start = time()
db.from_sequence((0.5 for n in range(100)), npartitions=100).map(sleep).compute()
print(time() - start, "seconds with", len(cluster.workers), "workers")
```
```
9.84409236907959 seconds with 3 workers
```

(Note that I set `scale_factor=1` to prevent anything but the target duration from scaling up the cluster.)

With single-threaded workers, we get what we desired (approx. 20s):
```python
[...]
cluster = LocalCluster(n_workers=1, threads_per_worker=1, memory_limit=1e9)
ca = cluster.adapt(minimum=1, maximum=10, target_duration="20s", scale_factor=1)
[...]
```
```
18.264188766479492 seconds with 3 workers
```

- Should this be documented?
- Or should behaviour of Adaptive be corrected to include the number of threads per worker in the estimate of the expected target duration?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.