kernelci / kernelci/kernelci-pipeline

Add support for tracking missing results

Open
#326 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11
Forks
40
Avg merge
2d 13h
Merged PRs (30d)
14

Description

When a test is not run or the result couldn't be sent to the API, for any reason, it would be very helpful to be able to report a warning. There are many reasons why this could happen, for example:

* The test has been removed from the test suite
* The test timed out
* The test crashed the system or the main program
* The system or the main program crashed before starting the test
* The results were in an invalid format
* The infrastructure failed (network, power, storage...)

Some of these cases are normal and expected, some are probably due to real kernel issues, some to test issues, some to infrastructure issues... Maybe with additional information we could refine the result, but by default this could be reported as a "warning" to the users.

Here's a proposed implementation:
* Before submitting a job, send nodes to the API for each expected test result (either loaded from some config or read back from the previous test run) with the result set to `null`
* When the job runs, nodes get updated with the results from each test as `pass`/`fail`/`skip`
* If any test wasn't run then its result will still be `null`
* Email reports, web dashboards etc. can interpret a node in the `done` state with a `null` result as a missing test case

Right now, the pipeline scheduler creates just one top-level node for each job and the job then sends a hierarchy of nodes for the results. The proposed implementation would mean creating a hierarchy with no results in the scheduler and updating the existing nodes in the job rather than creating new ones (which would lead to duplicates otherwise). The helper method and API endpoints to submit a hierarchy of nodes would need to be tweaked to look for existing nodes first and update any that has a matching path or create new ones otherwise. So the job logic itself wouldn't need to be changed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the pipeline scheduler's top-level job nodes, the job's hierarchy submission helper, and the API endpoints that create or update nodes. Define how expected tests are represented with null results and how matching paths are updated without duplicates. Done means unrun tests remain identifiable as missing while pass, fail, and skip results continue to report correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.