Concurent instances of bst should share cache without issues

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/585)
In GitLab by [[Gitlab user @valentindavid]](https://gitlab.com/valentindavid) on Aug 15, 2018, 12:33

## Background

[//]: # (Provide a background or the root/source that justifies this task or action.)

Currently we do not properly deal with concurrency in artifact cache.

For instance we could get the same issue #520, but while scanning removed artifacts.

Artifacts in cache should not be removed if still planned to be used by a task in a pipeline of any instance.

Also we should be able to add/remove artifacts/extracts concurrently without failure.

## Acceptance Criteria

[//]: # (Acceptance criteria should follow the S.M.A.R.T. principle https://en.wikipedia.org/wiki/SMART_criteria )

* Scanning artifact/extract directories can handle deletion of artifacts/extracts.
* Artifacts and extracts can be locked.
* Adding and removing of artifacts is safe to do concurrently (I believe it is already)
* Adding and removing of extracts is safe to do concurrently (I think it is not safe, we raise errors when we cannot commit extract directory).
* Calculating/caching the size of artifact cache should work across instances

----

No files or tests are named; start by reading the linked GitLab issue and locating the artifact-cache operations for scanning, locking, adding, removing, extracting, and size calculation. Done means concurrent BuildStream instances handle deletion and cache updates safely without failures, and cache-size calculation works across instances.

Written by the indexing model from the issue text.