Performance monitoring

[See original issue on GitLab](https://gitlab.com/BuildStream/buildstream/-/issues/205)
In GitLab by [[Gitlab user @samthursfield]](https://gitlab.com/samthursfield) on Jan 25, 2018, 09:43

Goals:

* avoid performance regressions
* provide feedback on proposed optimizations
* provide a single command which tests several versions of buildstream and outputs a set of results that can be analyzed
* allow running only a subset of tests

Components:

* [x] Project generation script/library
* [x] BuildStream log scraper -> script in contrib/ dir which parses BuildStream stdout and outputs a wellknown JSON format
* [x] Test harness which wraps project generation, obtaining multiple versions of BuildStream (using Docker images), and log generation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
* [x] Host machine analysis: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
* [x] Output analysis -- e.g. a table
* [x] Automation: https://gitlab.com/BuildStream/benchmarks/merge_requests/1
* [ ] Add documentation on how to run benchmarks in the CONTRIBUTING documentation

Configurable aspects:

* Scale of generated projects, e.g. 1 file, 10 files, 100 files ... lots of data points allow analyzing how a feature scales, but also means we have lots of data.
* Which test(s) to run

Out of scope initially:

* Simulation of low network speed / latency

We expect some of this to live in `contrib/benchmarking` inside the buildstream.git tree. The project generation tools and the test harness may as well live in a separate repo, at least initially.

Read the CONTRIBUTING documentation and inspect the benchmark components described in the issue, including contrib/benchmarking and the linked BuildStream benchmarks project. Document how to run the benchmarks, select a subset of tests, and configure generated project scale; the documentation is done when a contributor can follow it to run and analyze benchmark results.

Written by the indexing model from the issue text.