Feature request: Orquesta, with-items task and intermediate results availability

SUMMARY

To help further in migrating from Mistral to Orquesta the following example illustrates the future request:

      action_needed:
        with-items: action in <% $.config_data.actions %>
        action: st2autoremediation.remediation_actions
        concurrency: 1
        input:
          name: <% $.action.get('name') %>
          module: <% $.action.get('type', null) %>
          kind: <% $.action.get('kind', 'ansible') %>
          server_name: <% $.server_name %>
          params: <% $.action.get('params') %>
          stop: <% ('ERROR' in task(action_needed).state or 'cleared' in task(action_needed).result.subwf_result) or false %>
          filesystem: <% $.filesystem %>
          action_timeout: <% $.action.get('action_timeout', 60) %>
          as_root: <% $.action.get('as_root', false) %>
          disk_space_threshold: <% $.disk_space_threshold %>

This is a task which utilizes mistral's ability to provide intermediate results of the with-items task in the context and allows passing it as input parameters every time the action is called. With Orquesta it is currently not possible. It doesn't pass validation.

{
  "output": null,
  "errors": [
    {
      "message": "YaqlEvaluationException: Unable to evaluate expression '<% ('ERROR' in task(action_needed).state or 'cleared' in result().output.subwf_result) or false %>'. ExpressionEvaluationException: Unable to find task execution for \"action_needed\".",
      "type": "error",
      "route": 0,
      "task_id": "action_needed"
    }
  ]
}

The use of result() doesn't work either.

So the feature request comes down to either or both of the following:

1. Add a way to stop with-items iteration gracefully (based on some condition available every time action is actually being scheduled, including results from previous action's executions), or
2. Pass a value (again, based on the results already generated by the action) to the action being iterating over. (Use-case: there's no need to do anything anymore because previous iteration helped already, just run core.noop and finish.)

For item 1 there's even one improvement that comes into my mind: add a selector to stop it with success or stop it with error, so the correct transition could be taken afterwards.

If only item 2 gets implemented, then such selector can be implemented within the action itself based on the input values.

ISSUE TYPE

• Feature Idea

STACKSTORM VERSION

Paste the output of st2 --version:

st2 3.0.0, on Python 2.7.10

OS / ENVIRONMENT / INSTALL METHOD

CentOS 6, custom install.

STEPS TO REPRODUCE

So in case we mimic Mistral (item 2 from the summary), nothing changes in the task model per se. Just let task(task_name) and result() pass validation and if no execution has been run yet, return minimally available context, and result would probably be an empty list [].

If we add another field into the task model (item 1 from the summary), it could look like:

  action_needed:
    with:
      items: action in <% ctx().config_data.actions %>
      concurrency: 1
      stop:
        when: <% ('failed' in task(action_needed).status or 'cleared' in result().output.subwf_result) or false %>
        result: success # (or error), based on with what overall result we want to stop, but this not really critical to have
    action: st2autoremediation.nagios_disk_remediation_actions
    input:
      name: <% item(action).get('name') %>
      module: <% item(action).get('type', null) %>
      kind: <% item(action).get('kind', 'ansible') %>
      server_name: <% ctx().server_name %>
      params: <% item(action).get('params') %>
      filesystem: <% ctx().filesystem %>
      action_timeout: <% item(action).get('action_timeout', 60) %>
      as_root: <% item(action).get('as_root', false) %>
      disk_space_threshold: <% ctx().disk_space_threshold %>

@m4dcoder sorry for the late feature request submission!

Start with the task model and validation path for with-items workflows, then trace how task(action_needed) and result() are evaluated before any iteration has run. Compare the two proposed approaches—intermediate results for later iterations or a stop condition—and define completion as valid evaluation plus correct iteration and success/error behavior.