E2E Metrics

This is a new feature request.

Background:

We are spending a lot of effort in writing automation for every feature we develop. As the features become big and complex, the automation becomes more important. Internal teams, open source community and customers start to increasingly depend on automation for identifying whether a product release is well certified against a function or platform or something similar.

We need a mechanism or a tool to identify the E2E coverage of a feature or a release or an area in a time series such that a new process can be devised and followed around improving the quality of OEP.

Current status:

In OpenEBS.ci, for the community OpenEBS E2E, following process is followed:

• Folders are created based on platforms, as each pipeline is run against a platform
• In each platform folder, stages are created .. see for example https://github.com/mayadata-io/e2e-openshift/tree/master/Openshift-EE/pipelines/OpenEBS-base
• Each stage has multiple jobs.
• Each job is a test case.
• Each test case or job has a history of results of a test run on that platform with a pipeline id and the results with corresponding logs.

The above process is working. The main benefit of the above process seems to be to have the ability to go back in history of the pipeline runs and see the results of the same test case.

** Enhancements required: **

Following are some of the open questions that comes to my mind. We can try answering these questions and create small feature requests if needed or cover them in the design.

1. How to hierarchically represent a feature? Which is the parent feature or a component and child feature or a component?
2. How to get coverage of a given feature against a given platform or all platform?
3. How to get coverage of a given platform against one feature or all features?
4. How are we growing - Total number of test cases of a platform or a feature?
5. How many test cases are written in a given release or github milestone?
6. What is the coverage status of a given feature or a platform at the beginning of a release and end of milestone?

Keys of current metrics:

• Test case (UUID)
• Pipeline ID
• GitLab Job ID
• Platform ID
• Result (pass/fail)

Possible new keys:

• Time of test-run
• milestone
• Release
• Sprint
• Feature name
• Test case contributors