Slow execution time on very large k8s-backed dask cluster

**What happened**:

I'm trying to push the bounds of scaling kubernetes-backed dask clusters (using dask gateway), and the other day I started trying with a 4000-worker cluster, where 7 workers are packed into each node in our GCP cluster. I noticed that tasks started running ~1.5-2x slower than with a more reasonably sized cluster (couple hundred workers). This doesn't really surprise me at a high level - more workers means more time spent communicating with the scheduler, etc. But digging in a bit more, it is within the execution of the tasks themselves that the slowdown happens. I understand that communication could bog down the transmission of tasks through the scheduler, but I'm surprised that once a task begins executing, the task itself would take longer with a very large cluster.

**What you expected to happen**:
I expected tasks to execute in roughly the same amount of time regardless of cluster size and the time it takes for client-scheduler and scheduler-worker comms.

**Minimal Complete Verifiable Example**:
```python
import time

def test(x):
st = time.time()
for i in range(100000000):
pass
return time.time() - st

print(np.mean(client.gather(client.submit(test, range(4000))))

# ~1.9 on a 100-worker cluster
# ~3.75 on a 4000-worker cluster
```

**Anything else we need to know?**:
This is kind of a follow-on to my general "best practices for large clusters" issue #5164 (for which I know I owe a blog post...working on it 😬 ) But I figured starting a new issue that was specific to this particular scaling challenge was better than continuing to post to that thread

**Environment**:

- Dask version: 2021.09.1
- Python version: 3.8.12
- Operating System: Linux (running in a daskhub setup on GCP)
- Install method (conda, pip, source): conda