livepeer / livepeer/test-harness

Standardize test configuration

Open
#58 10 comments 0 reactions 1 assignee View on GitHub

@angyangie is already working on this.

Since May 10, 2019.

Dominant language
JavaScript
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

It would help to have a single configuration with known performance characteristics that can be reused across a variety of test cases. This gives us a few benefits:

* Better comparability across various test cases
* Reduced variability and improved reproducibility
* Minimizes guesswork when designing tests
* Avoids the combinatorial explosion in testing variations
* Better understanding of behavioral characteristics of the system

Here are some parameters to standardize on. Not all parameters will be applicable to all types of tests. This list is just a start, and not meant to be exhaustive:

* Transcoding options
* Number of segments to run the test over
* Number of concurrent streams
* Source stream and its characteristics
* VM instance types and configurations (`-maxSessions` etc)
* Number of nodes on the network and how much to provision per X streams
* Characteristics of the load. How quickly is it ramped up? How long does one stream last?

Often we may actually need to test variations on these standard parameters. That is totally OK and expected! By starting with a known standard configuration, we can keep other variables steady and isolate changes to only one parameter at a time. This allows us to see the effect of just that one parameter change.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.