dask / dask/distributed

Internalize keys and op-codes in bulk messages

Open
#7,386 6 comments 0 reactions 0 assignees View on GitHub
feature performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

# Rationale
Scheduler <-> Worker comms are typically dwarfed by Worker <-> Worker ones.

However, Scheduler <-> Worker comms may be a lot more expensive (in terms of time and/or money) than Worker <-> Worker ones too, so I think it's worth spending some effort in trimming them down.

Real life use cases:
- Coiled has the option to pick the "best available" AWS zone for workers, which is the zone that offers the most spot instances, (which are a lot cheaper than on-demand instances). In order not to double the cluster start time, however, it also needs to start the scheduler instances blindly in a different zone (since instance and worker instances are typically different). This causes worker and scheduler to end up in different zones, and **traffic between zones is paid for**.
- I recall a user starting a multi-region dask cluster, since they needed some workers to be local to their data. The scheduler would be in a different AWS region from said workers.

# Proposed design
Formalize that all batched comm messages must be `dict[str, Any]`.
Ahead of encoding them with msgpack, have a stateful (per-`BatchedSend`) mapping stage that finds and replaces all keys of the message, plus the value of the `op`, with incremental integers starting at -32 (because in msgpack ints in the [-32, 127] range cost only 1 byte). Every time a new string is encountered, it's stored and communicated in a special {-32: {-31: "str1", -30: "str2"} dict.

# Example

These worker->scheduler messages:
```yaml
- op: reschedule
key: x
- op: long-running
key: y
compute_duration: 123.4
- op: reschedule
key: z
```
Would be re-encoded into:
```yaml
- -32:
-31: op
-30: reschedule
-29: key
-31: -30
-29: x
- -32:
-28: long-running
-27: compute_duration
-31: -28
-29: y
-27: 123.4
- -31: -30
-29: z
```
Note how in the last message the -32 dict is omitted.

This whole extra encoding would be encapsulated in `BatchedSend` and invisible by both the worker and scheduler.

CC @ntabris @fjetter @gjoseph92 @hendrikmakait

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.