Improve compatibility of Buildstream with Buildbarn and Buildfarm

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/1114)
In GitLab by [[Gitlab user @Qinusty]](https://gitlab.com/Qinusty) on Aug 30, 2019, 11:13

## Background

Buildstream is currently not compatible with [Buildbarn](https://github.com/buildbarn), [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) or [RBE](https://console.cloud.google.com/marketplace/details/google/remotebuildexecution.googleapis.com?pli=1).

## Task description

Non exhaustive list of tasks

* [x] Merge https://gitlab.com/BuildStream/buildstream/merge_requests/1540 to support separate endpoints for remote caching and resolve https://gitlab.com/BuildStream/buildstream/issues/1050
* Address Buildbarn issues:
* [ ] [buildbarn-RE#22](https://github.com/buildbarn/bb-remote-execution/issues/22) Summary issue for Buildstream compatibility issues.
* Address Buildfarm issues:
* [ ] Investigate https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/43

## Discussion

With the goal of improving Buildstream compatibility with alternative RE server implementations aside from the tested [buildgrid](https://gitlab.com/BuildGrid/buildgrid). Buildstream requires the extension of `bst-artifact-server` to work with remote execution (RE) services which follow the [RE spec](https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/execution/v2/remote_execution.proto). This is fine as the overhead is not too large on the artifact server and https://gitlab.com/BuildStream/buildstream/merge_requests/1540 will allow for separate endpoints to be used for the CAS and index server instances.

### Buildbarn

Testing compatibility with Buildbarn is [detailed](https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/12) and it is clear that Buildbarn is [not following the spec](https://github.com/buildbarn/bb-remote-execution/issues/18). Buildbarn will have to correct the implementation of `output_directories` to be relative to the `working_directory` which is currently (correctly) relative to the `input root`. This is an unseen issue when Buildbarn interacts with Bazel as builds appear to specify the `working_directory` as `.` which is relative to the `input_root`. Following these changes, Buildstream should be compatible with Buildbarn given an appropriate image for sandboxing as detailed in [slack](https://buildteamworld.slack.com/archives/CD6HZC750/p1565275672010200).

```
Mickaël Carl 3:47 PM
--------------------
So indeed, bb-runner is currently provided on top of a Ubuntu image that’s also in use by RBE if I’m not mistaken
You are of course free to decide for yourself which base to use and move to a different base image

It’s actually incredibly simple, see https://github.com/buildbarn/bb-remote-execution/blob/master/cmd/bb_runner/BUILD.bazel

Specifically line 41
```

### Buildfarm

Buildfarm also causes issues while working with Buildstream however I am currently unsure of whether this an issue with Buildstream or Buildfarm. https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/43 details how to reproduce the issues I found and provides a stack trace pointing to an issue while uploading results. It is likely that Buildstream interacts with Buildfarm in a way that has gone unnoticed whilst Bazel has been used with Buildfarm.

----
[//]: # (To review information about possible relevant labels for this issue please view the list of labels: https://gitlab.com/BuildStream/buildstream/labels)

Start by reviewing the linked Buildbarn compatibility issue buildbarn-RE#22 and the remote-apis-testing issue 43, then compare their reproductions and stack trace with BuildStream's remote execution behavior. The work is done when the listed Buildbarn and Buildfarm compatibility problems are resolved and verified against the relevant remote execution services.

Written by the indexing model from the issue text.