Architecture: one program model, and composition-first growth

Motivation

Flowstate should be a small composition language over a frontend-neutral Protobuf IR and coherent runtime, gaining breadth through workflows, modules, plugins, and tooling—not through an ever-growing DSL.

Most of this shape already exists: Flowfile YAML+CEL compiles to flowstate.v1.Workflow; local and Temporal drivers interpret that specification; typed task descriptors feed validation, execution, LSP, MCP, and generated docs. The missing part is a concise platform model that makes the layer boundaries, replacement tests, and growth strategy explicit. Without it, a frontend feature can become an IR assumption, a Temporal mechanism can become runtime semantics, or an integration can become a new keyword when composition or a plugin would have sufficed.

Platform model and scope

Flowfile YAML+CEL   visual editors   alternate languages   SDK builders
         \              |                 |                  /
          +------------- frontend contracts ----------------+
                                |
                  normalized Protobuf IR (#1232)
                                |
                   semantic runtime (#1231)
                                |
       +------------------------+------------------------+
       |                                                 |
 local backend                                  Temporal backend
       |                                                 |
       +-------------- capability contracts ------------+
                    built-ins and plugins (#713)

Modules (#106) package and resolve reusable programs and supporting assets before normalization. Flowfile YAML+CEL is the first and canonical authoring frontend today, but it is not the platform. Temporal is the first-class durable backend, but it is not the semantic definition of a Flowstate program.

Each layer needs an explicit contract, owner, and conformance boundary so it can be tested and evolved independently. A layer is replaceable when another implementation can satisfy that contract and conformance suite without importing the replaced layer's private model; this does not imply runtime hot-swapping or equal operational properties.

• Frontends own source syntax, editions, source maps, author diagnostics, formatting, and migration into normalized IR.
• IR owns frontend-neutral, deterministic, typed program semantics and compatibility.
• Semantic runtime owns backend-independent execution meaning and observable state transitions.
• Backends own durability, clocks, scheduling, persistence, and capability realization under the runtime contract.
• Capabilities/plugins own typed effects and external integration contracts under deployment policy.
• Tooling projects the same descriptors, catalogs, diagnostics, and service contracts through CLI, API, LSP, MCP, docs, editors, and code generators.

Invariants

1. One schema-owned program model is the center of the platform; there is no YAML-shaped or backend-shaped second workflow model.
2. One semantic runtime interprets it; backends do not independently redefine nodes, failures, cancellation, or data flow.
3. Flowfile remains small. New breadth normally arrives through composition, modules, plugins, CEL, external services, and tooling (#234).
4. Local and Temporal execution agree where capabilities overlap; operational differences are explicit and Temporal remains first-class.
5. Every cross-boundary shape is Protobuf-first. Hand-written types add behavior or protect a value that must not serialize; they do not mirror contract data.
6. Descriptor and custom-option metadata describe one language-neutral fact once. Reflection and code generation may project it, but presentation and authority do not move into Protobuf (#729, #1227).
7. One capability catalog feeds validation, execution, CLI/API/LSP/MCP/docs, and generated SDKs; no surface maintains a second task or function vocabulary.
8. Policy, identity, secret handling, egress, isolation, provenance, and supply-chain admission fail closed at the boundary that owns the decision.
9. Work controlled by another party is bounded where it is spent.
10. Self-hosted local execution remains the baseline; optional infrastructure may add durability or reach, not redefine the language.

Breadth without language growth

Use the following order before proposing syntax:

1. Existing control-flow and data primitives.
2. CEL for pure, typed, bounded computation and policy.
3. Workflow composition and calls.
4. A module for reusable workflows, schemas, policy, helpers, docs, or capability bundles.
5. A plugin capability for an effect or protocol boundary the runtime should not own.
6. An external service reached through an existing capability.
7. New syntax only when the runtime must enforce a semantic distinction that none of the above can express or analyze.

A standard library should therefore be a governed set of small core primitives plus versioned, tested modules and official plugins. “Standard” describes trust, compatibility, documentation, and conformance—not a reason to place every useful integration in the grammar or engine binary.

Metadata and unified tooling

Protobuf descriptors, Protovalidate, CEL catalogs, and a minimal custom-options package (#729) should provide language-neutral contract facts. Source-bearing descriptor sets already support generated references, LSP hover, and MCP prose (#99/#424). The architecture should extend that derivation carefully across CLI, Connect API, LSP, MCP, docs, code generation, SDK builders, visual editors, and module/plugin catalogs while preserving these boundaries:

• schema metadata is descriptive and compatibility-relevant, never authorization by itself;
• mutating CLI/UI exposure is explicit and fail-closed, not “every request field becomes an input” (#1227);
• dynamic reflection is appropriate for bounded untrusted catalogs and read-only rendering; generated code is appropriate when build-time validation and static APIs matter;
• CEL implementations, cost, determinism, and policy scopes remain in one evaluator/runtime registry, even when metadata documents them;
• generated artifacts remain derived and drift-checked.

Composition, dependency, and trust model

Modules and plugins are different dependencies. A module contributes declarative content compiled into the IR; a plugin contributes runtime capability behind a process or remote trust boundary. A package may distribute both, but resolution and admission must preserve the distinction.

Dependency semantics must eventually cover identity, namespaces, imports/exports, visibility, version constraints, lock/pin behavior, transitive graphs, conflict rules, provenance, and reproducible offline resolution without prematurely choosing a registry. Local paths and repositories come first; future remote resolution remains content-addressed and compile/discovery-time, never mutable runtime linking.

Trust is not implied by discovery. Digests identify bytes; signatures/provenance may attest origin; policy decides admission; isolation bounds consequences. Module source, descriptors, custom options, generated code, plugin binaries, and catalogs are untrusted supply-chain inputs with bounded parsing, namespace/collision checks, dependency-cycle limits, provenance, and fail-closed security metadata (#146, #721).

Relationship to adjacent issues

• #234 — small DSL and composition-over-syntax invariant.
• #106 — module system and packaging semantics.
• #713 — language-neutral plugin capability contracts.
• #1232 — normalized frontend-neutral Protobuf IR.
• #1231 — backend-independent semantic runtime and conformance.
• #1226 — governed agentic development workflows composed on this platform, not a parallel orchestration stack.
• #172 — current compile-time workflow calls; #346 — ecosystem surfaces and module/tooling context; #336 — project roadmap map.
• #729/#1227 — custom-option and schema-derived interface boundaries; #425/#426 — documentation/tool delivery; #108/#241 — MCP/agent authoring surfaces.
• #239/#715 — capability taxonomy and plugin-system sequencing; #146/#151/#721 — integrity, remote locality, and isolation.

Non-goals

• Choosing a module registry, package file format, frontend syntax, remote transport, or plugin sandbox now.
• Making every layer replaceable at runtime; “replaceable” means a versioned contract and independent conformance, not hot-swapping an interpreter mid-run.
• Hiding Temporal or reducing Flowstate to the lowest common denominator of hypothetical backends.
• Turning Protobuf descriptors into UI configuration or policy authority.
• Filing a separate issue for every future standard-library module, frontend, backend, or tooling projection.
• Compatibility theater: Flowstate is super-alpha and may break, but durable artifacts, plugin contracts, and dependency identities still require deliberate migrations and honest versioning.

Open design questions

1. Which compatibility promises attach to each seam: source edition, IR semantics, runtime interpreter, backend adapter, module package, and plugin protocol?
2. What semantic conformance evidence makes a frontend, backend, module resolver, or plugin implementation supported?
3. Which metadata belongs in core descriptors, flowstate.options.v1, catalogs, or surface-owned configuration?
4. What is the minimum common dependency/provenance model for modules, plugins, generated SDKs, and future remote catalogs without forcing one registry?
5. How should core, official, blessed-module, and ecosystem tiers express compatibility and trust without self-asserted security labels?
6. Which existing package boundaries prevent the schema/IR/runtime layers from being independently consumable (#406)?

Acceptance criteria

• docs/ARCHITECTURE.md adopts the one-program-model diagram and names Flowfile as one frontend and Temporal as one first-class backend.
• Each layer has a documented contract, owner, allowed dependencies, compatibility boundary, and independent conformance strategy.
• Each seam has a reference fixture or test double proving an adjacent layer depends on the contract rather than a private YAML-, Go-, or Temporal-shaped model.
• #234, #106, #713, #1232, and #1231 settle their layer-specific designs without introducing duplicate evaluators, registries, schemas, or executors.
• A proposal checklist routes new breadth through primitives, CEL, composition, modules, plugins, or services before syntax.
• Descriptor/custom-option, reflection, code-generation, and CLI/API/LSP/MCP/docs ownership rules align with #729/#1227.
• Standard-library tiers and module/plugin ownership are documented without selecting a registry prematurely.
• Dependency, provenance, signature/digest, trust, isolation, and supply-chain responsibilities are mapped to their enforcing boundaries.
• One representative workflow can be produced by two frontends, normalized to the same semantics, and executed by local and Temporal backends with conformance evidence where capabilities overlap.
• #1226 and other architecture records link to this cluster rather than growing parallel platform models.

Start with docs/ARCHITECTURE.md and the linked issues #234, #106, #713, #1232, and #1231 to understand the proposed layer boundaries. Define the documented contracts, owners, dependencies, compatibility boundaries, conformance strategy, and seam fixtures described in the acceptance criteria. Done includes architecture documentation plus evidence that one workflow can be produced by two frontends and run by local and Temporal backends where capabilities overlap.