CQL Unit Testing

EDIT: Updated based on discussions below

The language server should support CQL unit tests. The first step is to define what a unit test looks like in CQL. We propose adding support for some @tags to support specifying unit tests and their input requirements in CQL

An example of a test Library using these tags:

// @test
// @parameter: "Measurement Interval" [@2019,@2020]
// @asof: @2020-10-01
// @context: Patient=654
// @terminology: tests/vocab
library DQMTest

include DQM as targetLibrary
include org.opencds.cqf.cql.Asserts version '1.0.0'
include TestHelpers

// @test
// @data: tests/data
// @context: Patient=123
define "In Initial Population":
  assert(targetLibrary."Initial Population").isTrue()
  
// @test
// @data: tests/data
// @context: Patient=123
define "Has Required Attributes":
   TestHelpers.HasRequiredAttributes(Patient)

Evaluating a definition marked with @test and getting a Message with the severity level of Error is a test failure.

http://cql.hl7.org/09-b-cqlreference.html#message

@test is required to use the other tags. If any are present without @test, it's an error condition. IOW, these tags are only allowed on unit test libraries.

Any @tag defined at the Library level sets the default @tag value for the set of tests within the Library. @tag values on a definition override the default Library @tag.

Each @test definition is evaluated independently and with the appropriate input parameters as defined by the merged definition and library @tag values. It's up to the test runtime environment to optimize data or context caching to speed up test execution.

If during the evaluation of a test, the CQL definition references any other CQL definitions that are also marked as @test (or with any other of the proposed tags), those @tags are ignored. Only the @tags of the entry point apply. That is to say, during the evaluation of a test the context, input parameters, terminology, or data may not change mid-evaluation.

CQL Test libraries SHOULD NOT be shipped as part of an executable package, such as a FHIR npm package.

The example org.opencds.cqf.cql.Asserts library does not exist at the time of this writing. The contents would be a set of helper functions to assist in specifying expected results for tests.

No implementation file or test entry point is identified. Start by clarifying the proposed CQL @test and related tag semantics, then determine how the language server and test runtime should represent and evaluate them. Done requires an agreed design and an implementation and test plan for unit-test libraries.