Observability as one system: traces, metrics, logs, and profiles that agree with each other and with Temporal

Umbrella for a research and design pass the owner asked for on 2026-08-13. Individual workstreams are filed separately and linked from here; this issue holds the goal, what exists today, and the invariants every workstream must respect.

The goal

A person debugging a production run should move between a trace, the logs for that run, the metrics that made them look, and a profile of the worker that ran it, without changing their mental model or re-deriving how the pieces relate. Today those are four systems that happen to be configured in the same process. The target is one system with four views.

That includes agreeing with Temporal rather than sitting beside it: a durable run's story spans Flowstate's own spans, Temporal's, the worker's activities, plugin processes, and outbound HTTP from the http task. If those do not join up, the trace is decorative at exactly the moment someone needs it.

What exists today, measured rather than assumed

• OTLP exporters for all three signals: otlptracehttp, otlpmetrichttp, otlploghttp.
• otelslog bridge, so log/slog output reaches the log pipeline.
• 12 metric instruments defined across the tree.
• baggage and attribute in use, so some propagation already happens.
• Telemetry setup lives in cmd/flow/telemetry.go, with a plugin-side half in pkg/flowstate/v1/plugin/telemetry.go and plugin/sdk/telemetry.go.
• A working demo stack in examples/observability/: compose file, Grafana, Loki, Tempo, Prometheus, an OTel collector config, plus a workflow and its test.

So the foundation is real. The open questions are coherence, coverage, containment, and whether any of it is actually verified.

Known gaps already recorded

• #401: telemetry testing covers logs over the wire and traces in a recorder, and metrics not at all.
• #422: sampling, sensitive-data containment, a stable attribute schema, and the profile story.
• #423: a ClickHouse golden path, one queryable store, tested.

Invariants every workstream must hold

1. One attribute schema. A run id, workflow name, step id, tenant, and trigger identity must be spelled the same way in a span attribute, a log field, and a metric label. Three spellings of the same concept is what makes correlation manual work.
2. Secrets never enter telemetry. The containment rules that keep secrets out of workflow history apply identically to spans, logs, metrics, and profiles. A span attribute is durable and broadly readable in exactly the way history is. Test the containment shapes, not just the value.
3. Cardinality is a bound, and the peer often controls it. Workflow names, step ids, trigger names and delivery ids are author or attacker chosen. An unbounded label set is a denial-of-service against your own metrics backend, so it gets bounded like every other resource here, and the bound is stated rather than hoped for.
4. Portability first, preference second. OTLP is the contract. ClickHouse is the owner's preferred store and Grafana the preferred frontend, but nothing may assume them: a deployment pointing at Tempo, Loki, Mimir, or a vendor must work, and the demo stack must not become the only tested path.
5. Both drivers. Local execution and durable execution must tell the same story about a run, or the local run stops predicting production.
6. Verified, not demonstrated. A dashboard that has never been asserted against real emitted data is a screenshot. A demo that only runs by hand rots. Whatever lands must be exercised by something that runs in CI.

Workstreams

Filed separately and linked here as they land: traces and propagation, metrics, logs and containment, continuous profiling and runtime debugging, and the backend plus dashboards plus demo story.

Read cmd/flow/telemetry.go, pkg/flowstate/v1/plugin/telemetry.go, and plugin/sdk/telemetry.go, then run the workflow and test in examples/observability/. This is an umbrella for separate traces, metrics, logs, profiling, and backend workstreams rather than a self-contained change. Done means the linked workstreams preserve the stated invariants and are exercised in CI.