Further remote execution testing

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/994)
In GitLab by [[Gitlab user @jennis]](https://gitlab.com/jennis) on Apr 12, 2019, 10:06

## Background

Now that we are testing remote execution in CI (https://gitlab.com/BuildStream/buildstream/merge_requests/1239), perhaps we should consider broadening our remote execution tests.

Currently we have two "simple" remote execution tests in [*tests/remoteexecution/simple.py*](https://gitlab.com/BuildStream/buildstream/blob/master/tests/remoteexecution/simple.py) which:

1. Build autotools with remote execution, check it out and verify it contains the right "stuff"
2. Build autotools remotely (again), then shell into it and execute `/usr/bin/hello` and ensure we get the right output.

This is a great start to remote execution testing in BuildStream but it far from comprehensive. I propose that we simply add some more tests.

## Task description

I'd imagine that there is an endless number of tests we could write for this, however, to start with I propose that we should add the following tests:

* [ ] Build *something* 'bigger' than autotools.
- The autotools example is great, however, why don't we try to build something a bit 'bigger', for example, a subset of freedesktop? Although I'd recommend avoiding the base ostree import for CI jobs as this is slow.
* [x] Build *something* via a junction.
- We already have a junction example in our docs. This just builds an element which depends on autotools via a junction, we could use this.
* [x] Build autotools with `bst --cache-buildtrees always build autotools/amhello.bst`
- This should mean that the buildtree of the artifact is locally cached (once built remotely), we should verify that it is there.
- We'd have to obtain the correct digest and then extract the *builddir* from the artifact.
- Some of our integration tests do something similar already.
* [x] Write an element that **will fail when built** and verify that this artifact contains a buildtree.
- Again, we'd have to obtain the correct digest and then extract the *builddir* from the artifact.

I'm open to any more suggestions, but I would like to avoid flooding this issue with so many additional tests that it never closes.

-----

## Aside: Running the remote execution test-suite locally

To run the remote execution tests, you need to have a buildgrid instance up and running, and at least one bot on port 50051.
Now, I personally did not find it trivial to get `tox -- --remote-execution tests/remoteexecution` to produce 'passing' tests. However, here is what my final setup looked like:

1. These tests require a couple of environment vairables: `export ARTIFACT_CACHE_SERVICE=http://localhost:11001 REMOTE_EXECUTION_SERVICE=http://localhost:50051`
- Note that I don't even think you need ARTIFACT_CACHE_SERVICE... but hey ho.
2. I had to slightly modify [Martin's Docker-compose manifest](https://gitlab.com/BuildGrid/buildgrid.hub.docker.com). To use these clone my fork: https://gitlab.com/jennis/buildgrid.hub.docker.com/tree/jennis/work_with_bst_tests and checkout the branch jennis/work_with_bst_tests

Now, you should be able to run the remote-execution tests, from one terminal:

docker-compose -f Composefile.buildstream.yml up --scale bots=2

Then, from another terminal, change to your BuildStream clone and:

export ARTIFACT_CACHE_SERVICE=http://localhost:11001 REMOTE_EXECUTION_SERVICE=http://localhost:50051
tox -- --remote-execution tests/remoteexecution

Alternatively, you can install BuildGrid and BuildBox fuse then start your own local buildgrid instance (with all the correct configuration...) and BuildBox (fuse) worker. Just follow the README instructions for both repos, but I found this difficult to get working.

Start with tests/remoteexecution/simple.py and the existing remote-execution integration tests, then review the tox -- --remote-execution tests/remoteexecution command and its required services and environment variables. The remaining unchecked goal is to add a test building something larger than autotools while keeping the remote-execution suite passing; avoid the slow base ostree import for CI.

Written by the indexing model from the issue text.