Python script evidence of input/output for validation.

When validating a python script during a Test Event or SVE, the python script does not provide supporting evidence to show that it is conforming to the test plan and does not provide evidence to support the reported verdict.

Description:

When reviewing the output from a python script test case there is a common theme that there is no log of the command that was issued by the test harness and no log of the returned value from a sample.

A script that was previously validated, but has been changed, excluding through the CCB process as this is slightly different, does not usually inherit its previously validated state under normal CSA rules. As a result, and following usual processes, these scripts become provisional again and need to be revalidated.

Example 1: python output.

A primary part of the requirements for a reviewer in a Test Event or SVE is to validate that the scripts going through testing are correct and the evidence for this is the log of that test case which is submitted for review. These logs are the primary evidence the CSA has historically used to show that test cases and test scripts are compliant and validated. The issue here is that the log does not provide evidence to show that the step has been run accurately and it does not show the values of attributes when read from a sample that would confirm the accuracy of the PASS/FAIL verdict presented by the script.
This means that a reviewer would not be able to confidently agree with the test or its verdict, either in an event or a live certification or a recertification review.

To satisfy the proof of evidence we would need the script output log to show the command issued from a test harness, such as with Example 2. and the response data from the sample that is assessed to generate the PASS/FAIL verdict, such as with Example 3.

Example 2: evidence of issued command.

Example 3: evidence of sample response to support a verdict.