picatz / picatz/flowstate

Plugin ecosystem: worked, CI verified examples — github, aws, gcp, terraform, k8s, git, slack, grpc, vault, docker, llm

Open
#95 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design examples kind/umbrella
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

The seam is closed: flow worker --plugin-dir registers plugin tasks into the registry every lookup reads, tasks are spelled <plugin>.<task>:, and the SDK plus a worked example exist under https://github.com/picatz/flowstate/tree/8915d29659707d54c1ade64b4bb9530612425aa5/pkg/flowstate/v1/plugin/examples/flowstate-plugin-example

Now build the ecosystem, each plugin landing with an example a Flowfile exercises in CI (the repo rule: a capability is not done until a Flowfile can express it):

Cloud and infrastructure plugins

  • aws: the flagship credential story — the outbound federation exchangers (#94) already speak AssumeRoleWithWebIdentity, so this is the plugin where JIT, audience-scoped credentials get proven end to end rather than configured as standing keys. Start with the calls runbooks actually make (s3 get/put, lambda invoke, sqs send, describe-instances-shaped reads) rather than attempting API breadth; the exchanger provides the session, the plugin makes the calls, and no long-lived credential exists anywhere in the chain.
  • gcp: same shape, same reason — the GCP STS + impersonation exchanger already exists in auth/exchange_cloud.go. Storage, pub/sub publish, and function invoke first.
  • terraform / opentofu: plan and apply as durable steps — plan produces an artifact a human approves through wait_for_signal:, apply consumes it after the gate. That pairing is the single most natural demonstration of what this engine is for (a plan/approve/apply pipeline that survives worker restarts and waits days for the approval), and it needs the exec/sandbox posture from #100 since it runs provider binaries. OpenTofu and Terraform are one plugin with a binary path config, not two.
  • kubernetes: apply/get/wait/delete against a cluster, for provisioning and operational runbooks that already live in kubectl scripts. Its own credential story (kubeconfig, in-cluster service account, or #94's federation) and its own policy surface (namespace allowlist, resource kind allowlist) — do not conflate with the docker plugin below, the two have almost nothing in common beyond both touching containers.
  • docker: run a container as a bounded, policy-governed effect — start, wait, capture output, tear down. Distinct from the sandbox-provider entry below: this is "run one container," not "provision an isolated execution environment."
  • sandbox provider (Modal or similar): a place to run untrusted work that is not the worker's own host and not a bare docker run; pairs with the exec policy work

Integration plugins

  • github: the full API surface runbooks and repo automation actually use — issues (create, comment, label, close), pull requests (open, review, merge, request changes), checks and statuses, commits and refs via the API, releases, and workflow-dispatch/run-status for Actions. This is the general-purpose GitHub plugin; two neighbors stay deliberately distinct: git below speaks the protocol (clone/diff/commit against any host — GitLab, Gitea, a bare remote), and the GHA trigger direction (a GitHub webhook starting a Flowstate run) is #96's webhook bridge, not a task. Credential story: a GitHub App installation token minted JIT — short-lived, permission-scoped, auditable as the app — with #94's federation as the eventual path to zero stored private keys; a PAT works locally and is never the documented production posture. Natural first example: triage automation — a workflow that labels an issue, requests review on a PR, and gates a merge behind wait_for_signal:.
  • git: clone, diff, commit for repo driven workloads — protocol-level, host-agnostic; not to be folded into the github plugin above
  • slack and discord: post now, signal bridge later so an approval gate is answered from the channel that asked
  • grpc: the generic unary caller; streaming rows stay refused until they have an execution model
  • vault / openbao: the canonical secrets plugin example (in tree secrets/vault is the in process ancestor); blocked on #91 for the resolution path
  • 1password: exists in tree, needs local verification against a real agent before it is claimed
  • llm: provider agnostic completion and tool calls, the base of agentic workloads

Observability-stack plugins (the deliberately circular set, see #98)

  • temporal: query and act on the Temporal cluster the workflows run on (describe, terminate, reset, search) from inside a workflow
  • grafana: file annotations, query dashboards, raise or resolve alerts
  • loki: query logs, so a workflow can read the log lines its own run produced
  • prometheus / tempo as they earn a use: query metrics and traces, closing the loop where a workflow reads its own telemetry

The circularity is the point when threaded carefully, and it must stay optional: none of these is required to run a workflow, and the observability lab in #98 is where they are proven together.

steps:
  - id: plan
    terraform.plan:
      dir: ${steps.checkout.dir}
  - id: approve
    wait_for_signal:
      name: apply-approved
      timeout: 72h
  - id: apply
    if: ${!steps.approve.timed_out}
    terraform.apply:
      plan: ${steps.plan.artifact}
  - id: release
    github.release:
      repo: acme/api
      tag: ${vars.version}
      notes: ${steps.plan.summary}
  - id: notify
    slack.post:
      channel: "#infra"
      text: ${'applied ' + steps.plan.summary}

Positioning note from docs/VISION.md: the built in registry stays small (log, http, exec once its policy exists); breadth lives here. Security posture (sandboxing, JIT access, #100) is the gating concern for anything that executes code or holds credentials, not a follow up — the aws/gcp plugins are specifically where the #94 federation work stops being infrastructure and becomes a user-visible capability, and the github plugin's App-token JIT posture is the same principle applied to the forge.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read docs/VISION.md, auth/exchange_cloud.go, and the worked example under pkg/flowstate/v1/plugin/examples/flowstate-plugin-example; related work includes #94, #96, #98, #100, and #91. The issue spans many plugins and security policies rather than naming one bounded change. Done requires each selected capability to have a Flowfile example exercised in CI, with execution and credential concerns resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, gcp, git, github, github-actions, go, grafana, grpc, kubernetes, prometheus, terraform
Domain
api, ci-cd, cloud, devops, infrastructure, observability, security, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.