Enable knowledge of standardized results in PaC to support decisions on non-failing PipelineTasks

In situations where there are many related tasks in a single PipelineRun, I want to be able to have all tasks run "successfully" so that the entire pipeline will be run even if some tasks are not strictly successful. While it makes sense to have some PipelineRuns actually fail (for example, if an artifact build fails), there are other types of tasks that might successfully run while also reporting issues (for example, test or vulnerability scans). In addition to formalizing on a result name, we can also formalize on the minimum data reported in these results so that PaC would be able to return consistent data.

While there would be a standard definition for PaC, Tekton/PaC should not strictly enforce the fields to enable further extension of the result types by tasks as needed.

Proposal for standardizing on test-type tasks

Tekton result

TEKTON_TEST_RESULT

Required fields

• result - ENUM containing values SUCCESS, FAILURE, WARNING, SKIPPED or ERROR
• successes - the number of successes returned in a test
• failures - the number of failures returned in a test
• warnings - the number of warnings returned in a test

Expected handling

result value(s)	GitHub status conclusion value
SUCCESS	success
SKIPPED	skipped
FAILURE, ERROR	failure
WARNING	neutral

Sample output

{
    "result": "SUCCESS",
    "timestamp": "1649843611",
    "successes": 16,
    "note": "An example of an extra field ignored by PaC",
    "failures": 0,
    "warnings": 0
}

Proposal for standardizing on vulnerability scan-type results

Tekton result

TEKTON_SCAN_RESULT

Required fields

Following the convention of security ratings: https://access.redhat.com/security/updates/classification/, the following results should be supported

• critical
• high
• medium
• low

Expected handling

PaC should return the result of the most severe non-zero result in the Checks table.

In order to enable users to change the Checks conclusion based on the presence of vulnerabilities, PaC can enable a customization for a minimum_allowed_scan_result. If this parameter is specified, then a maximum result at or below the configured value would correspond to a conclusion of success and any value above would correspond to a conclusion of failure. If no value is set, all results would return a conclusion of neutral.

Sample output

{
  "critical": 0,
  "high": 1,
  "medium": 1,
  "low":0
}

Would be handled with a maximum result of high.

No implementation files, tests, or entry points are named. Start by locating how PaC currently interprets Tekton results and produces GitHub Checks conclusions. Done means standardized test and scan results are recognized, configurable conclusions are produced as proposed, and extra result fields remain supported.