dask / dask/distributed

Double counting of network transfer cost

Open
#7,003 3 comments 0 reactions 0 assignees View on GitHub
diagnostics performance scheduler scheduling stealing
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

We're double counting estimated network cost in multiple places

Forst, we're calculating the estimated network cost of dependencies a worker needs to fetch in [`_set_duration_estimate`](https://github.com/dask/distributed/blob/b133009cee88fd48c8a345cffde0a8e9163426a6/distributed/scheduler.py#L2784-L2810) and are setting the result to `WorkerState.processing`, i.e. `processing = compute + comm`
This is also used to set the workers **occupancy**

When making a scheduling decision, we're typically using `Scheduler.worker_objective` which calculates a `start_time` that is defined as

https://github.com/dask/distributed/blob/b133009cee88fd48c8a345cffde0a8e9163426a6/distributed/scheduler.py#L3000-L3001

i.e.

```python

start_time = ws.occupancy / ws.nthreads + comm_bytes / self.bandwidth
= ws.occupancy / ws.nthreads + comm_cost

occupancy ~ sum( ... TaskPrefix.duration_average + comm_cost )

```

1. comm cost should be constant and not scale with nthreads
2. we should only account for comm_cost once

A similar double counting is introduced on work stealing side when calculating the cost_multiplier

```python
compute_time = ws.processing[ts] # occupancy
transfer_time = nbytes / self.scheduler.bandwidth + LATENCY
cost_multiplier = transfer_time / compute_time

# If we ignore latency for now, this yields something like

cost_multiplier ~ NBytes / (Bandwidth * duration_average + NBytes)
= (NBytes / Bandwidth) / (duration_average + NBytes / Bandwidth)
```

i.e. for network heavy tasks, this converges towards 1 which is quite the opposite of what this ratio is supposed to encode

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.