Improve metrics being exposed to prometheus

I've been futzing around with monitoring Dask with prometheus a little lately (some blog posts are in the pipeline).

I wanted to open an issue to discuss exposing more metrics via the `/metrics` endpoint.

Currently we expose the following:

- Scheduler
- Number of clients
- Number of workers and their states (`connected`, `saturated`, `idle`)
- Desired number of workers
- Forgotten tasks
- Tasks and their states (`released`, `no-worker`, `waiting`, `memory`, `erred`, `processing`)
- Workers
- Tasks and their states (`stored`, `executing`, `waiting`, `ready`, `serving`)
- Number of connections to other workers
- Latency of worker connections
- Number of threads

Given that it is common to run additional exporters such as the [node-exporter](https://github.com/prometheus/node_exporter) I don't think we need to worry about system type metrics, but there are other things I can think of that might be nice to expose here. I would also love input from others if you can think of things.

- Managed vs unmanaged worker memory (based on the work @crusaderky has been doing in #4634/#4651)
- Some proxy for the task stream data. Perhaps something like percentages of time spent processing, communicating, swapping, idle. Perhaps something like [this](https://github.com/dask/distributed/issues/4260#issuecomment-738183697) from @Timost in #4260.