Developer documentation

In an off-line discussion about technical debt and code complexity the valid concern was raised that many of our internal systems are not properly documented.

One example that came up is the current/new state machine (https://github.com/dask/distributed/issues/4413 https://github.com/dask/distributed/pull/5046) which is documented to some extend (https://distributed.dask.org/en/stable/scheduling-state.html and https://distributed.dask.org/en/stable/worker.html#internal-scheduling) but likely not sufficiently for another developer to make educated judgment calls about code changes.

I would like to collect topics, mostly for `dask/dask` and `dask/distributed` where more extensive developer documentation would help either onboarding new developers or help existing developers to familiarize themselves with other areas of the code.

cc @jcrist @jrbourbeau @gjoseph92 @ncclementi

- [x] https://github.com/dask/distributed/issues/5413
- [ ] https://github.com/dask/distributed/issues/5414
- [ ] https://github.com/dask/distributed/issues/5415
- [ ] https://github.com/dask/dask/issues/7755
- [ ] https://github.com/dask/distributed/issues/5416
- [ ] https://github.com/dask/distributed/issues/5417