Bazel clean sometimes fails with Directory not empty

### Description of the problem / feature request:

`bazel clean` on my CI workers sometimes fails with the following error:

```
ERROR: unlinkat(/var/lib/buildkite-agent-ci/.cache/bazel/output_base/...) (Directory not empty)
```

It is really difficult to reproduce because the next invocation of `bazel clean` does succeed most of the time. I do actually run `bazel clean || bazel clean || bazel clean` in my CI as a workaround.

I'm using a custom `--output_base` shared between multiples bazel workspace (called `output_base` here).

There is no other bazel process running on the worker at the same time.

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I'm sorry, I have no idea. I'm opening the bug report in order to document my issue. I'll try to add more information if they become available.

### What operating system are you running Bazel on?

Linux

### What's the output of `bazel info release`?

`release 3.3.1- (@non-git)`

### If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

Using bazel from nixpkgs.

### Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/5907 is similar, but is fixed since years and was apparently only on Windows.

Start by investigating the reported Linux failure when running bazel clean with a custom shared --output_base; the issue provides no source files or tests to begin with. Reproduction is currently unavailable, so done would require identifying the cause and making bazel clean reliably remove the output base without a Directory not empty error.

Written by the indexing model from the issue text.