Path towards improved testing (especially end to end testing)

The following is my attempt to understand the testing needs of Spin and sketch a possible path towards improved end to end testing for Spin.

What needs testing

Before we can improve testing we need to know what actually is under test:

Host Runtime

• Loading and linking of any WebAssembly binaries built with any version of the Spin SDK or the fermyon:spin@2.0.0 world.
• Host implementations for all Spin interfaces (both the unversioned Spin world used by Spin 1.x SDKs and the fermyon:spin@2.0.0` world).
  • Including any changes to host implementations that might happen due to flags being set through the Spin CLI.
• The correct component export is invoked with the correct inputs and the expected outputs are observed.
• The Spin runtime produces a "locked application" that can be understood by any Spin compliant runtime.
• All versions of the manifest are successfully parsed and result in the runtime being configured correctly.
• The Spin CLI error messages (i.e., that when issues occur the correct output is displayed to the user).

Guest Components

• The SDKs for all supported languages produce WebAssembly binaries that can be linked to and loaded by the Spin runtime.

Spin related tooling

Additionally there are other Spin related tools that exist outside of producing Spin apps and running them that also need to be tested.

• Spin doctor
• Spin plugins
• Spin watch

The status quo

Currently, Spin testing generally merges many of the above concerns into one single test meaning when a test fails it's often hard to know why. For example, the end-to-end tests not only test that an SDK (usually the Rust SDK but sometimes Go) successfully builds a WebAssembly binary, it then runs that binary against the current Spin runtime and checks for outputs. Which outputs are checked are relatively ad-hoc, and it's therefore very easy to introduce new functionality in the Spin runtime and not test this functionality.

On the guest side, we currently don't test the Python or JavaScript SDKs at all as part of CI and testing against the Go SDK is spotty at best.

Testing system requirements

Any testing system we create needs to fulfill the following needs:

• Aim for testing as close to 100% as possible of the functionality listed at the start of this document.
• Aim to test as much of the functionality as possible in isolation only integrating when there is a chance that the integration itself might break.
• Allow for sharing of tests for common functionality.
  • We might want to test some functionality across many different implementations (e.g., two SDKs that both target the same Spin runtime but are for two different languages or two different Spin compliant runtimes). We should make as much of the testing framework easily shareable across implementations as possible only hard coding the Spin runtime located in this repo when the functionality under test is explicitly specific to this runtime (e.g., error messages).
• Make tests run as fast as possible.
• Make adding tests as easy as possible
• Make it easy to see where there are gaps in test coverage
• Ensure that functionality tested by separate tests overlaps as little as possible (i.e., if a piece of expected behavior changes, we should minimize the number of tests that need to change).
• Make test failures show clear and understandable reasons for failure to ensure quick fixes
• Make the testing harness as simple and easy to change as possible.

Possible steps forward

I unfortunately don't have a grand unified vision of exactly how testing should be in the future, but hopefully this document can guide us as we continue to evolve the system. That being said, there are some items that seem like they might be good steps forward for replacing the e2e tests with something more sustainable:

• Replace e2e tests with a test suite that works against pre-built WebAssembly components.
  • These components would live in their own repo (perhaps pulled in by the Spin repo as a submodule) which could be used by other Spin runtimes for their testing purposes.
  • Having pre-built components means that running the tests should be much faster as the component does not need to be built
  • This also breaks the testing dependency between the SDK and the runtime so that the runtime can be tested independently of the SDK
  • These pre-built components could be written in many languages or all in the same language as the language used to create them is not really a concern to the runtimes being tested. In fact, we may wish to create these components without using an SDK.
• Create a "test runtime" for testing SDKs
  • Given that the Spin runtime can be tested independently of the SDKs, we can use a "test runtime" that expects the same shape of component as the Spin runtime but only tests that inputs into the runtime and results back from the runtime get translated appropriately. For example, when testing key-value#set, we wouldn't test that the value is actually set in a store, just that the value passed to the set function from the SDK under test correctly makes it to the test runtime and that the SDK sees that correct return value.
  • This means that we don't ensure the full e2e story for SDKs and the runtime (which has proven hard and error prone to do in practice), we just ensure that the SDK conforms to the correct component contract and we use the runtime tests described above to ensure that the runtime can execute any component that conforms to that contract. This moves testing from an M:N problem (i.e., M SDKs against N runtimes) to two N:1 problems (i.e., 1 test component to N runtimes and 1 test runtime to N SDKs).
  • This would also make testing SDKs much more systematic as the test runtime would test all imports meaning that implementors of new functionality only need to update the test runtime and all SDKs would then receive test coverage. Each SDK would then either implement the new functionality to make the test pass or explicitly opt-out (perhaps because no one is currently available to implement the functionality in some language's SDK). This allows for a much easier overview of the state of each SDK.

Start by reviewing Spin's current end-to-end tests and CI coverage for the runtime and supported SDKs, as described in the issue. Compare the existing approach with the proposed pre-built WebAssembly components and test runtime; done means an agreed, maintainable testing strategy is implemented with clearer coverage and failure boundaries.