Scheduler ignores current resource consumption levels while scheduling

Currently on the scheduler, when a task is assigned to a worker and consumes resources, that's set in one place. When deciding whether a task can be assigned to a worker, that's checked in a different place. Therefore, current resource consumption levels are not considered in task scheduling.

The current scheduling appears to just consider which workers can run a task _in theory_: do they have enough of the resource to be able to run this task ever (even if none of it is available right now)?

Considering resources like GPUs, I suppose this makes sense: queuing extra tasks onto workers is beneficial so there's no idleness. Still, it's a little surprising. And the fact that `worker_objective` doesn't take current resource consumption into account seems likely to cause bad scheduling, since we could easily assign a task to a worker whose resource is currently used up, when there are other workers with the resource available.

----

When a task gets assigned to a worker, `consume_resources` only adjusts the count in `WorkerState.used_resources`:
https://github.com/dask/distributed/blob/e0ea5df5bc56b130f66d2741d9bc24bfa1eb0121/distributed/scheduler.py#L2674-L2675

But `SchedulerState.valid_workers` looks for which workers can run a task, it only checks `self.resouces[resource][address]`, and never looks at `WorkerState.used_resources`:
https://github.com/dask/distributed/blob/e0ea5df5bc56b130f66d2741d9bc24bfa1eb0121/distributed/scheduler.py#L2644-L2652

So tasks will not [enter the `no-worker` state](https://github.com/dask/distributed/blob/e0ea5df5bc56b130f66d2741d9bc24bfa1eb0121/distributed/scheduler.py#L1759-L1766) just because all resources in the cluster are currently used up.

Instead, as usual, more tasks will get queued onto workers than they can run at once. Each worker will manage only running the correct number of tasks at once.

----

* Is this intentional?
* Why do we track resource counts in both `self.resources` and `WorkerState.resources`?
* Why do we bother tracking `WorkerState.used_resources` if it's never actually used for scheduling decisions?

Note that changing this behavior would likely provide a viable temporary solution for https://github.com/dask/distributed/issues/6360, a very common pain point for many users.

cc @mrocklin @fjetter