Connecting to a non-existent client on the dashboard port doesn't time out

**Describe the issue**:
When accidentally trying to (incorrectly) connect a client to dashboard port instead of scheduler port, the operation doesn't timeout like before.

**Minimal Complete Verifiable Example**:

```python
from distributed import Client, LocalCluster
from dask import config

if __name__ == "__main__":
cluster = LocalCluster(dashboard_address="0.0.0.0:8788")
with config.set({"distributed.comm.timeouts.connect":3}):
c = Client("localhost:8788") # hangs
print(c)
```

**Anything else we need to know?**:
The client connection times out as expected for other ports except the one the dashboard is set on.
I've explicitly set the dashboard address here to show it's somehow related to that, but keeping the defaults and trying to connect to `8787` also fails.

I was able to bisect it to 2751741721d3f2bee636b4281952ed0c9405cfeb. Prior to this, the operation would time out as expected.

**Environment**:

- Dask version: Dask@2023.7.1+9.gbdb827b7, Distributed@2023.7.1+10.g27517417
- Python version: 3.10
- Operating System: ubuntu20.04
- Install method (conda, pip, source): pip from source