Feature Request - Load/store data/artifacts/binaries from external content source

SUMMARY

Currently in a distributed StackStorm deployment, when running an action the node that the action is run on is random. This causes some headaches when trying to deal with files or artifacts when implementing things like a ETL workflow or CI/CD workflow.

ETL:

• Usually requires processing large amounts of data
• Usually queries and persists data between multiple databases

CI/CD:

• Usually works with local files
• Usually produces some sort of "build artifact" (RPM, DEB, Docker container, etc)

The way this works now is:

ETL (Database query)

• Write one action to query the database (SQL, NoSQL, etc)
• Next action would take the output of the query as input and do the transform
• Final action would take the transformed data and persist it back to the database
• Problem: Data can be very large and saving this in Mongo can add a large tax on the workflow processing time

CI/CD (Files and Binaries)

• Write a action(s) that create a build artifact(s) (RPM, DEB, Docker container, etc)
• Write another set of actions that upload these build artifacts to a repository (Artifactory, Nexus, S3, etc)
• Problem: The data is too large to passed between tasks, so we have to write it to a common NFS share and pass a file path between the the steps.

ISSUE TYPE

• Feature Idea

IDEAS

Another workflow tool that i found has an interesting concept of Artifacts that it can be passed between steps in the workflow:

• https://github.com/argoproj/argo/blob/master/examples/README.md#artifacts
• https://github.com/argoproj/argo/blob/master/examples/README.md#hardwired-artifacts

This spawned some thinking and relates to an idea i had in: https://github.com/StackStorm/st2/issues/4343

It would be cool if we could pass in "artifacts" as inputs/outputs associated with a task in a workflow. The task would perform some pre/post work to load/store the artifact around the action run.

Sudo coding it could look something like what i had in my other request.

ETL - Database

This would retrieve a database artifact from Mysql, do some processing, then publish the results back to Mysql.

vars:
  sql_connection: "{{ st2kv.system.sql_connection }}"
  
tasks:
  task1:
    action: transaction.place_orders
    input_artifact:
      mysql:
        connection: "{{ ctx().sql_connection }}"
        # by default this returns a list of dicts
        query: "SELECT id,name,date FROM orders ORDER BY date DESC;"
    input:
      data: "{{ input_artifact().mysql.result }}"
    next:
      - when: "{{ succeeded() }}"
        publish_artifact:
          mysql:
            connection: "{{ ctx().sql_connection }}"
            insert:
              # name of the table
              table: "history"
              # list of dicts to insert
              values: "{{ result() }}"

CI/CD - Files and Binaries

This would run a build process that checks out a git repo, builds the thing, uploads the RPM to a Yum repo and uploads the build log to an S3 bucket.

vars:
  sql_connection: "{{ st2kv.system.sql_connection }}"
  
tasks:
  build:
    action: cicd.build
    input_artifact:
      git:
        # downloads the repo to a local path on the actionrunner
        repo: https://github.com/org/repo.git
    input:
      path: "{{ input_artifact().git.path }}"
    next:
      - when: "{{ succeeded() }}"
        publish_artifact:
          nexus:
            path: "{{ result().rpm_path }}"
            upload: rpm
            url: "{{ st2kv.system.nexus.rpm_upload_url }}"
            username: "{{ st2kv.system.nexus.username }}"
            password: "{{ st2kv.system.nexus.password | decrypt_kv }}"
          s3:
            path: "{{ result().build_log_path }}"
            endpoint: storage.googleapis.com
            bucket: my-bucket-name
            key: path/in/my/bucket
            accesskey: "{{ st2kv.system.s3.accesskey | decrypt_kv }}"
            secretkey: "{{ st2kv.system.s3.accesskey | decrypt_kv }}"

Reusing existing packs

Ideally it would be great if packs could plugin to this "artifact" architecture and provide input/output artifact actions that could be run. This would allow us to have pluggability and not reinvent the wheel or have to pull in code complexity for integrations within StackStorm core itself.

Long story short, this is just a cool thing i saw and wanted to write down my thoughts / usecase before i forgot it.

No implementation files or tests are named. Read issue #4343 and the linked Argo artifacts examples first, then trace StackStorm workflow and action handling; this issue needs a concrete artifact design, scoped integration points, and agreed acceptance criteria before implementation can begin.