Double counting of network transfer cost
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
We're double counting estimated network cost in multiple places
Forst, we're calculating the estimated network cost of dependencies a worker needs to fetch in [`_set_duration_estimate`](https://github.com/dask/distributed/blob/b133009cee88fd48c8a345cffde0a8e9163426a6/distributed/scheduler.py#L2784-L2810) and are setting the result to `WorkerState.processing`, i.e. `processing = compute + comm`
This is also used to set the workers **occupancy**
When making a scheduling decision, we're typically using `Scheduler.worker_objective` which calculates a `start_time` that is defined as
https://github.com/dask/distributed/blob/b133009cee88fd48c8a345cffde0a8e9163426a6/distributed/scheduler.py#L3000-L3001
i.e.
```python
start_time = ws.occupancy / ws.nthreads + comm_bytes / self.bandwidth
= ws.occupancy / ws.nthreads + comm_cost
occupancy ~ sum( ... TaskPrefix.duration_average + comm_cost )
```
1. comm cost should be constant and not scale with nthreads
2. we should only account for comm_cost once
A similar double counting is introduced on work stealing side when calculating the cost_multiplier
```python
compute_time = ws.processing[ts] # occupancy
transfer_time = nbytes / self.scheduler.bandwidth + LATENCY
cost_multiplier = transfer_time / compute_time
# If we ignore latency for now, this yields something like
cost_multiplier ~ NBytes / (Bandwidth * duration_average + NBytes)
= (NBytes / Bandwidth) / (duration_average + NBytes / Bandwidth)
```
i.e. for network heavy tasks, this converges towards 1 which is quite the opposite of what this ratio is supposed to encode
Contributor guide
Assessment
This issue has not been assessed yet.