Big Feature: Workflow Testing

This is a super high level ticket to capture an idea for working testing. I don't even know if this is a kit issue - although it's probably a CLI feature (lightining might want it to)

I'm using the word "project" here to mean "file system around a workflow" - because I don't think these are workflow.json concerns, strictly speaking. I'm talking about new assets to support a workflow.

The Problem

A very typical pattern in workflow creation is:

• The client will provide some sample input data and some expected output data (the output may be in a third party system, like salesforce)
• Mutchi will create a workflow in the CLI and built it to match this output data
• After a bit of wriggle with the client, the workflow will run as intended and can be tested on a wider sandbox system before going out for deployment

It would be really cool to be able to formalise the initial input/output stages into a suite of integration tests. To make them part of the project and to be able to run openfn test and it'll run a suite of tests based on input and output specs.

The Other Problem

Within a workflow, there are often gnarly or complex functions which are used in certain transformations. Date and data mappings for example.

These functions are tested at the workflow level - but it would be nice, for confidence and code quality and even for devx - to be able to unit test them standalone. This would ensure the functions work as expected with certain inputs and outputs. This greatly increases the confidence of the function within the workflow.

So it would be neat to associate a suite of unit tests with a project.

Solutions?

openfn test

This command will run any tests found in the target folder. We'd have a convention like .test.js or something for all test files, and find them with a test running. This might be formally associated with a project or workflow, but probably it'll just be files in the same folder.

This formalises and automates the process of manually checking the output of a workflow. It means the test is portable and can run on different user systems, and a range of tests cases/suites can be added to add robustness and increase confidence in the workflow.

Tests will need a structure something like this:

const input = { data: {}, configuration: { user: 'abc', password: 'password1' } }
const output = { data: [ /* array of patients */ ] }
test(workflow, input, output, options)

We could probably use file conventions for data-driven tests, rather than using code.

We might find that to facilitate this, workflows have to be more strict about creating upload data on state in a transformation phase, and doing a simple upload in the adaptor phase. Like this:

// transform.js (common)
fn((state) => {
  // transform fhir data to salesforce
  state.sfupload =state.data.map(...)
return state;
})

// upsert.js (salesforce)
upsert((state) => state.sfupload)

This would allow tests to test the transformed data, not the actual upload step.

Related to that, the tests are likely to want to ignore some steps in the workflow. In fact you probably just want to test the transformation steps and none of the data service steps.

We might want the test to include some fuzziness. For example if the transformation adds timestamps or dates, we don't need those dates to match exactly. So we'll have to work out how to add some tolerance - maybe fuzzy paths (if both paths are truthy and the same data type, they're considered close enough).

One difficulty is that we already have an openfn test command. Maybe it's time to retire it. Or maybe it'll run the test command if you don't point to a directory. So openfn test . will run tests in the current folder, and openfn test will run the built-in test job.

Unit Tests

The above describes integration tests - testing a workflow and validating its output state.

Unit tests - the testing of a specific function within the workflow - are also desirable.

But unit tests are really hard because the function we want to test is probably declared in an fn() block. It may or may not be written to state. This makes it hard to access and test from outside the workflow.

We may need to have a better way of declaring a re-usable function within the workflow (this should be spun out into a separate issue).

Unit tests would probably be written with a library like mocha or ava and use the regular js expect/assert patterns. I don't think we need to do anything formal or interesting here - we just need to:

• Allow a workflow to define a callable function (maybe it can just export the function from the job? Actually I think this is supported out of the box?)
• Allow the CLI to use a test runner like ava

Lightning Integration

Which, if any, of these tests should be supported by Lightning?

Should a Project have a Tests tab which lets you define integration and unit tests? How would they be executed? Through the worker I guess? But you'd have to clone the project repo or something in the worker.

Start by reading the existing openfn test command and its current built-in test behavior. Clarify whether the scope is workflow integration tests, standalone unit tests, Lightning support, or a smaller subset, then define the test-file conventions and execution model. Done should mean an agreed, implementable testing workflow rather than a high-level proposal.