Adaptive target duration is scaled by `threads_per` worker
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Adaptive Clusters with `threads_per_worker > 1` will scale to reach a duration `target_duration / threads_per_worker`:
```python
from dask.distributed import LocalCluster, Client, Adaptive
from dask import bag as db
from time import time, sleep
cluster = LocalCluster(n_workers=1, threads_per_worker=2, memory_limit=1e9)
ca = cluster.adapt(minimum=1, maximum=10, target_duration="20s", scale_factor=1)
client = Client(cluster)
start = time()
db.from_sequence((0.5 for n in range(100)), npartitions=100).map(sleep).compute()
print(time() - start, "seconds with", len(cluster.workers), "workers")
```
```
9.84409236907959 seconds with 3 workers
```
(Note that I set `scale_factor=1` to prevent anything but the target duration from scaling up the cluster.)
With single-threaded workers, we get what we desired (approx. 20s):
```python
[...]
cluster = LocalCluster(n_workers=1, threads_per_worker=1, memory_limit=1e9)
ca = cluster.adapt(minimum=1, maximum=10, target_duration="20s", scale_factor=1)
[...]
```
```
18.264188766479492 seconds with 3 workers
```
- Should this be documented?
- Or should behaviour of Adaptive be corrected to include the number of threads per worker in the estimate of the expected target duration?
Contributor guide
Assessment
This issue has not been assessed yet.