UCX cluster shutdown errors when GIL contention monitoring enabled

**Describe the issue**:
When shutting down a UCX cluster with GIL contention monitoring enabled (i.e. `gilknocker` is installed and `distributed.admin.system-monitor.gil.enabled=true`), we get some worker errors of the form:

```python
2023-05-01 12:46:05,381 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/home/nfs/charlesb/dev/distributed/main/distributed/comm/ucx.py", line 349, in read
await self.ep.recv(msg)
File "/datasets/charlesb/mambaforge/envs/distributed-gpuci-py310/lib/python3.10/site-packages/ucp/core.py", line 725, in recv
ret = await comm.tag_recv(self._ep, buffer, nbytes, tag, name=log)
ucp._libs.exceptions.UCXCanceled: <[Recv #183] ep: 0x7fd3f40730c0, tag: 0x42adb052da7b4792, nbytes: 16, type: >:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/nfs/charlesb/dev/distributed/main/distributed/worker.py", line 1237, in heartbeat
response = await retry_operation(
File "/home/nfs/charlesb/dev/distributed/main/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/home/nfs/charlesb/dev/distributed/main/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/home/nfs/charlesb/dev/distributed/main/distributed/core.py", line 1269, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/home/nfs/charlesb/dev/distributed/main/distributed/core.py", line 1028, in send_recv
response = await comm.read(deserializers=deserializers)
File "/home/nfs/charlesb/dev/distributed/main/distributed/utils.py", line 754, in wrapper
return await func(*args, **kwargs)
File "/home/nfs/charlesb/dev/distributed/main/distributed/comm/ucx.py", line 367, in read
raise CommClosedError(
distributed.comm.core.CommClosedError: Connection closed by writer.
Inner exception: UCXCanceled("<[Recv #183] ep: 0x7fd3f40730c0, tag: 0x42adb052da7b4792, nbytes: 16, type: >: ")
```

**Minimal Complete Verifiable Example**:

In an environment with UCX-Py and gilknocker:

```python
from distributed import LocalCluster

cluster = LocalCluster(protocol="ucx")
cluster.close()
```

**Anything else we need to know?**:
These worker errors are seemingly the cause of / directly related to errors that cropped up in the UCX tests running on GPU; I've circumvented the tests for now by manually disabling GIL contention monitoring for GPU tests:

https://github.com/dask/distributed/blob/4bd2ba7ec4e4a4669c13dc1e62820191664783f4/continuous_integration/gpuci/build.sh#L26-L27

But ideally would be nice to get to the root of this issue and remove this override.

**Environment**:

- Dask version: 2023.4.1
- Python version: 3.10
- Operating System: ubuntu
- Install method (conda, pip, source): source