Error when using client for DaskXGBClassifier

Hi,
I've ran into a problem when I was trying to use XGBClassifier for dask within a Docker image [see here also](https://github.com/dmlc/xgboost/issues/7546).
### Summary
Using:
`client = Client(n_workers=4, threads_per_worker=1, processes=False, memory_limit='16GB', scheduler_port=0, silence_logs=False, diagnostics_port=0) `
And after fitting with:
`model.fit(X_t, y_t, eval_set=(X_e, y_e)) `
I get:
` model.fit(
File "/usr/local/lib/python3.8/dist-packages/xgboost/dask.py", line 1817, in fit
return self._client_sync(self._fit_async, **args)
File "/usr/local/lib/python3.8/dist-packages/xgboost/dask.py", line 1617, in _client_sync
with distributed.worker_client() as client:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/dist-packages/distributed/worker_client.py", line 55, in worker_client
duration = time() - thread_state.start_time
AttributeError: '_thread._local' object has no attribute 'start_time'
distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/5
distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/4
distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/6
distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/7
distributed.scheduler - INFO - Remove worker
distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/5
distributed.scheduler - INFO - Remove worker
distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/4
distributed.scheduler - INFO - Remove worker
distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/6
distributed.scheduler - INFO - Remove worker
distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/7
distributed.scheduler - INFO - Lost all workers
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms`

**Environment**: docker image - ubuntu-latest

- Dask version: 2021.12.0
- Python version: 3.9.9
- Operating System: MAC OS Big Sur 11.6.1
- Install method (conda, pip, source): pip