opengeospatial / opengeospatial/ogcapi-processes

Is a first-class "Workflow" resource, distinct from Process, in scope for Part 3?

Open
#633 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Part 3 (Workflows/Chaining)
Dominant language
AsciiDoc
Stars
63
Forks
46
Avg merge
2d 1h
Merged PRs (30d)
8

Description

Coming from the workflow-orchestration side (I maintain pygeoapi-prefect, integrating pygeoapi with Prefect), it stands out to me that Part 3 is called 'Workflows and chaining' and yet it does not include the concept of a Workflow as a first-citizen entity.

This seems contrary to the design taken by most mature orchestration tools, regardless of domain, in which it is common to have both the concept of a workflow definition (named, versioned, independently addressable/listable) distinct from individual processes and that of a workflow run (an execution instance), distinct from its individual job runs). Some examples:

Tool Definition Run
Prefect Flow Flow Run
Apache Airflow DAG DagRun
Argo Workflows Workflow (k8s CRD, own UID/status) (execution is the resource's own status)
AWS Step Functions State Machine (ARN) Execution (own ARN)
GitHub Actions Workflow (.yml) Workflow Run
Nextflow Workflow (.nf) Run
Cylc Workflow (flow.cylc) Flow instance

To be precise about terms: by Workflow here I mean a directed acyclic graph of process executions - the output of one process invocation feeding into another as input, however many processes and steps may be involved.

This is an orchestration concept that sits above any single process execution unit - it's about which Process resources get invoked, in what order, with what data flowing between them.

It is a different thing from a single process' execution unit internally being implemented as a multi-step pipeline (e.g. a CWL Workflow with several Steps packaged into one container) — that's a Part 2/execution-unit implementation detail, invisible to an OGC API caller as anything other than one process with one execution endpoint.

As far as I can tell, Part 3 currently has exactly one mechanism that expresses this DAG-of-processes concept at all, and it has no persistence:

Ad-hoc/ephemeral, via Nested Processes: an execute request can chain already-deployed processes together, an upstream process's output becoming a downstream process's input (see PR #607, still open, for the current CWL-based ad-hoc proposal). This forms a real DAG of process executions for the duration of one request, but it has no identity or persistence beyond that request, nothing that could be searched, re-run by reference, list prior runs of, or version later.

There have also been proposals to persist such a DAG so it can be deployed once and re-invoked without re-specifying the whole chain each time (e.g. the DeployWorkflow proposals discussed in #431, which use $input/$output references to compose already-existing processes). But per #431, every proposal there converges on the same outcome: the persisted result is a new resource at /processes/{processId}, a plain Process. The DAG-of-processes composition collapses into a single process identity, indistinguishable at the API level from a Process backed by an ordinary single-step execution unit.

I couldn't find anywhere that the identity/run-history question was posed on its own terms, independent of how deployment or execution-unit packaging works. So I'm posing it now:

Was a first-class Workflow resource (distinct from Process, with its own identity and run-history) considered and rejected for Part 3?

If it was considered, a pointer to that discussion would be helpful. If not, it seems worth a dedicated issue rather than being addressed implicitly through the deployment-mechanics threads above.

I don't have a proposal for how such a resource should be designed - it just seems like a notable omission, given that most workflow engine tools have this concept in their models.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Part 3 and the nested-process proposal in PR #607, then review the DeployWorkflow discussion in issue #431. Determine whether a first-class Workflow resource with separate identity and run history is in scope, and document the relevant prior decision or the design questions that remain.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.