kernelci / kernelci/kernelci-pipeline

Investigate support for multiple runs

Open
#278 0 comments 0 reactions 0 assignees View on GitHub
roadmap
Dominant language
Python
Stars
11
Forks
40
Avg merge
2d 13h
Merged PRs (30d)
14

Description

Some tests may need to be run several times with exactly the same combination of parameters to gather additional results and produce statistics. This is particularly useful for tests results that aren't stable or to get an average with performance runs. The paths and all the meta-data will be the same for each run, except some fields that will be used to distinguish results that are coming from the same run.

For example, we could have an "iteration" field for this which would be automatically incremented every time an identical data set is being sent.

Here's a typical scenario with first the data being sent to the API:
```json
{
"name": "foobar",
"path": ["parent", "foobar"],
"result": "pass",
"data": {
"score": "1234"
},
}
```
```json
{
"name": "foobar",
"path": ["parent", "foobar"],
"result": "pass",
"data": {
"score": "1432"
},
}
```

Then the data stored in the database would add `"iteration": 1` to the first entry and `"iteration": 2` to the second one. One issue with this is that the client can't determine the iteration value when submitting the data, so the data read back from the API is not the same as what was sent.

Another approach is to not implement anything, several nodes with the same parent and same path and same attributes can be retrieved with a query as a list. So the default approach for now would be to see whether the existing data model can already cope with this or if we need some additional features. This should become more apparent when trying to use the data to compute stats or detect performance regressions etc.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the existing API data model and how nodes with the same parent, path, and attributes are stored and retrieved. Check whether repeated submissions can already be returned as a list, then investigate how the data would support statistics or performance-regression detection. Done means documenting whether existing behavior is sufficient or defining the additional iteration features required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.