Request for feedback: Restructuring the environment object

As creator of a HIL (Hardware in the Loop) system for designers and manufacturers of hardware with their own software I very much value your effort in bringing CTRF forward. It in most places is a very good fit without getting unnecessarily complex.

But now to the part of the CTRF where I hope for some improvement:

In section "10. Environment Object" the spec states ([n.] added by me for easier referencing):

The environment object describes the execution environment [1.], system configuration [2.], and CI/CD context [3.] in which the test run was executed.

Comments:

1. "Execution environment" makes it completely clear, that the contents within the environment object is about the environment the software/app/service/device under test (xUT in short) was executed.
2. "System configuration" is a little broad and unspecific (which probably is intended). Most importantly it does not contradict "execution environment". So, fine but maybe superfluous?
3. "CI/CD context" is also clear. From the traceability viewpoint, CI/CD information is very valuable. The only question I'd like to raise is if this is part of the "execution environment" or more an artifact of how the xUT was built.

In the following I distinguish between the following:

• "test execution environment" (aka "execution environment" [1.])
• "build execution environment" (e.g. CI/CD context [3], the host that executes a make file, ...)
• "software/app/service/device under test (xUT)": the system, artefact or entity that is tested

Now to the single items (with "extra" omitted):

• section "10.1. reportName":
  Has nothing to do with the environment, maybe move to top level to its cousin reportId?
• section "10.2. appName" and "10.3. appVersion":
  • The xUT's name and version: understood
  • Is the xUT part of the environment?
• sections "10.4. buildId", "10.5. buildName", "10.6. buildNumber" and "10.7. buildUrl":
  • The ID/name/number/url generated by the build execution environment when it built the xUT for later identification.
  • This actually could be seen as property of the xUT.
• sections "10.8. repositoryName" and "10.9. repositoryUrl":
  The spec says "[...] source code repository associated with the test run.".
  • Please clarify: Is it about the xUT's repository or about the test environments repository?
  • I assume the test environment and tests are also under version control.
  • Repository information make sense for all of them.
• section "10.10. commit":
  The spec says "The commit hash or revision identifier of the code under test.".
  • explicitely xUT related
  • not about the test execution environment
  • Is the xUT part of the environment?
  • It seems more to be a cousin of "9.27. device" (reslts.tests[].device).
• section "10.11. branchName":
  The spec says "[...] branch from which the test run was executed."
  • Please clarify: Is it about from which branch the xUT was built by the build execution environment or about the branch the test execution environment is from?
  • I assume the test environment and tests are also under version control.
  • Branch information make sense for all of them.
• sections "10.12. osPlatform", "10.13. osRelease" and "10.14. osVersion":
  • the test environments OS related information; understood so far
• section "10.15. testEnvironment":
  • test execution environment related
• section "10.16. shardId":
  • test execution environment related
• section "10.17. healthy":
  • test execution environment related
  • The part "the overall system or" more confuses me than clarifying things.

For traceability reasons it would make sense to collect:

• repositoryName
• repositoryUrl
• commit
• branchName
• osPlatform, osRelease, osVersion

for each of:

• the "test execution environment"
• the "build execution environment"
• the xUT itself

separately.

So the question arises, what to include under "environment" (in the sense of test execution environment) or broaden its coverage officially.

Start by reviewing section 10, “Environment Object,” and the listed fields from reportName through healthy. Determine whether the specification should separate test execution environment, build execution environment, and the xUT, and document the agreed placement and meaning of repository, build, source, and host details.