Support workflow versioning

Background

Proposed @opensearch-project/opensearch-ux mockups for the frontend plugin introduces a concept of launches within a single workflow. This allows for easy prototyping for a particular use-case, allowing users to construct and test different workflow configurations, persisting the history and associated resources with each deployment for reference.

The current backend design does not support this paradigm. One approach to support this is introducing versioning to workflows, opening up functionality to iterate on a workflow at the API level to persist and provision multiple versions of a single workflow. Originally discussed in #109, this issue expands on this idea.

Implementation

Currently there is a version field (see here) but it is not being used or validated yet. We can refactor this into being a required field with a String value that users can set to whatever they want. This way, there will always be a default version for a workflow. (If we need to persist compatibility, we can refactor that into a standalone template field)

Internally, workflows with the same ID but different versions can behave identically to the current workflow implementation; users can perform CRUD operations on them, they each persist their own provisioning state, created/associated resources, etc.

To handle defaults at the API level, backend logic can filter through the workflow ID, and if there are multiple instances of that workflow ID under different versions, choose the one with the latest last_update_time or equivalent such field.

API changes

Users need the capability to run CRUD operations on individual versions of a workflow if they want. Users who don't want/care about versioning shouldn't have to, with abstracted out logic and sufficient defaults. Most changes listed below are just adding an additional path to existing APIs that include a version ID.

Create workflow

Users specify a (now mandatory) version field in the template.

POST /_plugins/_flow_framework/workflow
{
   "version": "my-version-1",
   "name": "my-workflow-name",
   ...
}

[NEW] Create new version of a workflow

Same as create, but now include the already-created workflow-id in the path. The JSON body should include a non-name-clashing version with any existing versions for that workflow, else throw an error. See open questions below on what to include in this JSON body.

POST /_plugins/_flow_framework/workflow/<workflow-id>
{
   "version": "my-version-2",
   "name": "my-workflow-name",
   ...
}

Update workflow

Same as before, with a new optional version-id to specify which version of the workflow to update. If no version specified, default to the latest version

PUT /_plugins/_flow_framework/workflow/<workflow-id>
PUT /_plugins/_flow_framework/workflow/<workflow-id>?version-id=<version-id>

Get workflow

Same as before, with a new optional version-id to specify which version of the workflow to get. If no version specified, default to the latest version.

GET /_plugins/_flow_framework/workflow/<workflow-id>
GET /_plugins/_flow_framework/workflow/<workflow-id>?version-id=<version-id>

Provision workflow

Same as before, with a new optional version-id to specify which version of the workflow to provision. If no version specified, default to the latest version

POST /_plugins/_flow_framework/workflow/<workflow-id>/_provision
POST /_plugins/_flow_framework/workflow/<workflow-id>/_provision?version-id=<version-id>

Get workflow state

Same as before, with a new optional version-id to specify which version of the workflow to get state. If no version specified, default to the latest version

GET /_plugins/_flow_framework/workflow/<workflow-id>/_status
GET /_plugins/_flow_framework/workflow/<workflow-id>/_status?version-id=<version-id>

Deprovision workflow

Same as before, with a new optional version-id to specify which version of the workflow to deprovision. If no version specified, default to the latest version

POST /_plugins/_flow_framework/workflow/<workflow-id>/_deprovision
POST /_plugins/_flow_framework/workflow/<workflow-id>/_deprovision?version-id=<version-id>

Delete workflow

Same as before, with a new optional version-id to specify which version of the workflow to delete. If no version specified, default to all versions

DELETE /_plugins/_flow_framework/workflow/<workflow-id>
DELETE /_plugins/_flow_framework/workflow/<workflow-id>?version-id=<version-id>

No changes/updates needed for the Get Workflow Steps or Search Workflow APIs.

Open questions

1. For "Create new version of a workflow", should the JSON body be a subset of what's created on the initial workflow creation? For example, we probably don't want name / description / use_case overridden by some later version. Fundamentally these fields should be static. Internally maybe we propagate such static fields to the newly-versioned workflow when indexing.
2. Performance concerns. Most APIs will need added logic to fetch a specific (or all) version(s) of a workflow under a workflow ID.

(ASIDE) Frontend implications

This approach seamlessly fits into the frontend design for handling multiple launches/deployments/versions. Matching up UX action vs. API calls:

1. Creating new workflow = create workflow API with some default (or user-specified) version
2. Viewing all workflows = search workflow API and deduping by workflow ID (constructing aggregate query and filtering by latest last_update_time)
3. Clicking "launch" = provision with current version ID (if first launch), or create new version of a workflow with provision=true flag (if subsequent launch)
4. Viewing all launches = search workflow API and parsing out all versioned workflows that match the workflow ID
5. Viewing launch resources = get workflow state API with specified version ID + workflow ID
6. Prototyping launch = get workflow API with workflow + version ID to prototype with (how to prototype is an open question - new API vs. using search pipelines stored under its resources vs. other)

Updates

3/5: updated example API paths to have version-id be query param instead of in the path itself

Start by tracing the existing workflow CRUD, provisioning, status, and deletion API entry points and the workflow template's version field; the issue does not name implementation files or tests. Resolve the open questions about static fields and default/latest-version behavior, then add coverage for version-specific and default API operations.