`Failed to link` - `file exists` on cache pull

While using buildstream I found the following issue:

```
[--:--:--] STATUS [59065303] self_junction_a.bst:example.bst: Pulling data for artifact 59065303 <- https://...
[--:--:--] INFO [59065303] self_junction_a.bst:example.bst: Pulled artifact 59065303 <- https://...
[--:--:--] INFO [59065303] self_junction_b.bst:example.bst: Pulled artifact 59065303 <- https://...
[00:00:13] SUCCESS [59065303] self_junction_a.bst:example.bst: Pull
[00:00:14] FAILURE [59065303] self_junction_b.bst:example.bst: Failed to link '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189 -> .../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f': [Errno 17] File exists: '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189' -> '.../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f'
```

In my project I use multiple self-junctions - junctions that load the project itself to build it differently or to overwrite some elements.
While pulling artifacts in parallel for different self-junctions from cache there's this issue `Failed to link ... File exists`.

I cannot reproduce this error locally, only on specific CI.

I found mentions of the same error in existing issues/PRs:

* https://github.com/apache/buildstream/issues/1740
* https://github.com/apache/buildstream/pull/1881#issuecomment-1837236850
* https://github.com/apache/buildstream/pull/1820#issuecomment-1426807683

My guess is that its some race caused by [`safe_link`](https://github.com/apache/buildstream/blob/4c92f3003cfba379edd9d653a72d860ded013445/src/buildstream/utils.py#L387-L398) function.

I'm trying to produce some fix for this race, what should I consider while tinkering with `safe_link`?
My best bet is to make link operation more atomic by first making hard-link in some temp directory and them moving it to original destination.

Start with safe_link in src/buildstream/utils.py, especially lines 387-398, and review the referenced issue 1740 and pull requests 1881 and 1820. Exercise parallel cache pulls involving multiple self-junctions, then determine the expected behavior when concurrent linking targets the same reference. Done means the race is handled without a spurious File exists failure.

Written by the indexing model from the issue text.