kernelci / kernelci/kernelci-pipeline
Add support for tracking missing results
- Dominant language
- Python
- Stars
- 11
- Forks
- 40
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 14
Description
When a test is not run or the result couldn't be sent to the API, for any reason, it would be very helpful to be able to report a warning. There are many reasons why this could happen, for example:
* The test has been removed from the test suite
* The test timed out
* The test crashed the system or the main program
* The system or the main program crashed before starting the test
* The results were in an invalid format
* The infrastructure failed (network, power, storage...)
Some of these cases are normal and expected, some are probably due to real kernel issues, some to test issues, some to infrastructure issues... Maybe with additional information we could refine the result, but by default this could be reported as a "warning" to the users.
Here's a proposed implementation:
* Before submitting a job, send nodes to the API for each expected test result (either loaded from some config or read back from the previous test run) with the result set to `null`
* When the job runs, nodes get updated with the results from each test as `pass`/`fail`/`skip`
* If any test wasn't run then its result will still be `null`
* Email reports, web dashboards etc. can interpret a node in the `done` state with a `null` result as a missing test case
Right now, the pipeline scheduler creates just one top-level node for each job and the job then sends a hierarchy of nodes for the results. The proposed implementation would mean creating a hierarchy with no results in the scheduler and updating the existing nodes in the job rather than creating new ones (which would lead to duplicates otherwise). The helper method and API endpoints to submit a hierarchy of nodes would need to be tweaked to look for existing nodes first and update any that has a matching path or create new ones otherwise. So the job logic itself wouldn't need to be changed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the pipeline scheduler's top-level job nodes, the job's hierarchy submission helper, and the API endpoints that create or update nodes. Define how expected tests are represented with null results and how matching paths are updated without duplicates. Done means unrun tests remain identifiable as missing while pass, fail, and skip results continue to report correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100