Randomly at the end of a program I get an error that some temporary directory cannot be removed

**What happened**:

I get the following error randomly at the end of execution of a script using dask (it does not happen all the time):

```
distributed.scheduler - INFO - Receive client connection: Client-6ca31a72-1620-11ec-81a8-20000fa5fe80
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Run out-of-band function 'stop'
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
distributed.worker - INFO - Stopping worker at tcp....:36164
....
distributed.diskutils - ERROR - Failed to remove 'work/dask-worker-space/worker-acf10xla' (failed in ): [Errno 2] No such file or directory: 'work/dask-worker-space/worker-acf10xla'
distributed.wo
```

**What you expected to happen**:
When a temporary directory is not found, I would expect dask not to try to remove it and not fail...

**Anything else we need to know?**:

I am running the script in a job, so when it exits with an error, the job also fails.

I use `initialize` function to start dask workers from the python script:

```python
if args.mpi:
from dask_mpi import initialize
initialize(nthreads=1,
# dashboard_address=None
)

```

And then create a client using context:

```python
with Client() as client:
# do work
```

**Environment**:

- Dask version: 2021.08.0
- Python version: 3.8.1
- Operating System: Linux
- Install method (conda, pip, source): pipn with local cluster OpenMPI