Client.cancel may block for up to one second

If a key is cancelled, the scheduler currently waits for up to one second for the key to arrive. This is implemented to guard us from a race condition where a key was scheduled and immediately cancelled but the cancellation arrives on the scheduler before the compute message.

This waiting period was introduced in https://github.com/dask/distributed/pull/6016 where this wait was instead implemented by scheduling a background task, i.e. retry after Xs. Not blocking can cause a race condition if there are multiple clients interested in the same key as shown in #6016

https://github.com/dask/distributed/blob/bd98e66039a38851ea33295a927255dcd5319601/distributed/scheduler.py#L4299-L4305

I believe this wait could be avoided if we ensured that the client cancellation request would similarly move over the stream comm since then ordering would be guaranteed and we could return the cancellation immediately if there is no key.