Conformance Testing

As more and more tooling is ramping up, we're going to need some sort of conformance tests to allow tooling to validate that their implementation of Arazzo is correct. It is especially important so that someone working with one tool to create the Arazzo description and another to execute the description can expect it to work as intended. Especially as more and more tools add execution capabilities, it will be imperative that descriptions execute in the same manner of determinism.

These tests should account for every scenario including the upcoming Async support in 1.1 and future support for actor-in-the-loop, function calling, mcp, etc. Naturally it may take time to build a strong repository of conformance tests and the tests themselves should be descriptive in the scenario they test and the version(s) of Arazzo supported as well as versions of OpenAPI, AsyncAPI and future source description formats.

We can start off with very simple conformance tests but the goal is to cover a wide range of possible workflows to maximize the conformance coverage.

The issue does not name files, tests, or an entry point. Start by locating the project's existing test and specification structures, then define a small initial conformance scenario; done should include descriptive tests that identify supported Arazzo and source-description versions and validate deterministic workflow behavior.