Formalize conformance test description format

Discussed on TSC Meeting: April 2, 2018:

Rules and test cases in a conformance test suite might have a descriptive structure, something like this:

• Rule description (human-readable)
• Reference to the relevant part of the spec.
• Set of test cases derived from this rule:
  • Example input
  • Expected result, as one of the following:
    • pass
    • error (corresponds to MUST, MUST NOT in the spec)
    • warning (corresponds to SHOULD, SHOULD NOT in the spec)
  • Comments (optional)

OpenAPI implementations, including editors, generators, documentation formats, etc., are expected to validate the content, correctly detect error and warning conditions (without false-positives on valid content), expose and handle errors and warnings in whatever way is appropriate for that implementation.

Some goals, and possibly non-goals, pending discussion:

• Create a definitive conformance test suite as the set of all of these rules and test cases.
• Use a uniform descriptive structure for the test cases in the suite, something like the preceding bullet list.
• Make the rule and test case descriptions machine-readable.
• Make the rules machine-executable by adding structured representations of each rule, which could take different forms:
  • Where the rule can be expressed in JSON Schema, include it in a schema, which can be designed to be the standard schema for OpenAPI, or could be given a more limited scope of use, as an informative addition to the conformance test suite.
  • In other cases, or possibly in all cases, state the rule as a logical assertion using an expression language, preferably one that can be embedded or interpreted in different programming languages.

Consider all of the above as a straw-man proposal, for discussion.