Proposal for Multi-Library, Multi-Resource Tests

This is based off of a discussion we had at the spring 2024 HL7 Connectathon.

The group discussed at length about adding the ability to run tests that support multiple libraries as well as accepting input resources. Since the current tests only support single statements we cannot currently test features like valuesets or the code in valueset feature. Other features like fluent functions would also be benefitted by this feature.

The team discussed a potential format for defining a test group that overcomes these and other existing gaps and came up with the following example json. The following is an example meta file for defining the inputs to a test runner which would be needed to run one of these tests and also like the associated file outputs to be asserted on.

testGroupName.json

{
  "description": "...",
  "cqlLibraries": [
    "testFile.cql"
  ],
  "resources": [
    "patient.json"
  ],
  "valuesets": [
    "valueset.json"
  ],
  "outputFile": "output.json"
}

For the output file we as a team discussed using the json definition defined here https://github.com/cqframework/clinical_quality_language/wiki/CQL-Serialization for the expected output format.

Start by reviewing the proposed test-group JSON in the issue and the linked CQL Serialization definition. Identify the test-runner entry point and existing single-statement test format, then determine how completion would be verified for multiple CQL libraries, input resources, valuesets, and asserted output files.