livepeer / livepeer/test-harness
Standardize test configuration
@angyangie is already working on this.
Since May 10, 2019.
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
It would help to have a single configuration with known performance characteristics that can be reused across a variety of test cases. This gives us a few benefits:
* Better comparability across various test cases
* Reduced variability and improved reproducibility
* Minimizes guesswork when designing tests
* Avoids the combinatorial explosion in testing variations
* Better understanding of behavioral characteristics of the system
Here are some parameters to standardize on. Not all parameters will be applicable to all types of tests. This list is just a start, and not meant to be exhaustive:
* Transcoding options
* Number of segments to run the test over
* Number of concurrent streams
* Source stream and its characteristics
* VM instance types and configurations (`-maxSessions` etc)
* Number of nodes on the network and how much to provision per X streams
* Characteristics of the load. How quickly is it ramped up? How long does one stream last?
Often we may actually need to test variations on these standard parameters. That is totally OK and expected! By starting with a known standard configuration, we can keep other variables steady and isolate changes to only one parameter at a time. This allows us to see the effect of just that one parameter change.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.