observability: verify the demo stack and dashboards instead of parsing them (workstream of #522)

Workstream of #522 (backend, dashboards, demo story). Companion to #423, which already
scoped the ClickHouse golden path and the integration-test tier for it. This issue is
the piece #423 leaves as "Related: #401" and folds in invariant 6 (verified, not
demonstrated) as its own concrete mechanism, plus where each check belongs in CI. It
does not redo #423's design; it narrows on what #423 left open and adds the dashboard
half.

1. The architectural point, checked against the current tree

Invariant 4 says OTLP is the contract and nothing may assume ClickHouse or Grafana.
That holds today. cmd/flow/telemetry.go builds otlptracehttp, otlpmetrichttp,
and otlploghttp exporters against OTEL_EXPORTER_OTLP_ENDPOINT and stamps
semconv.ServiceName/ServiceVersion — nothing in that file, or in
pkg/flowstate/v1/plugin/telemetry.go, names Tempo, Loki, Prometheus, or ClickHouse.
examples/observability/otel-collector/config.yaml is where the backend choice
actually lives: otlp/tempo, otlphttp/loki, and prometheus exporters, wired only
in the collector config, which is exactly the seam invariant 4 asks for. Swapping a
backend today means editing that YAML, not the binary. Confirmed, not assumed: routing
lives in the collector, the app only speaks OTLP.

2. What the existing demo stack actually proves today

Read in full: docker-compose.yaml, README.md, otel-collector/config.yaml,
tempo/tempo.yaml, loki/loki.yaml, prometheus/prometheus.yml,
grafana/provisioning/datasources/datasources.yaml,
grafana/dashboards/flowstate.json, workflow.yaml, workflow.test.yaml.

Bluntly: the compose file proves it parses. CI runs

docker compose -f examples/observability/docker-compose.yaml config -q

which needs no daemon, pulls no images, and validates YAML shape and cross-references
(volumes, service names) — it would catch a broken anchor, not a broken pipeline. The
README says this plainly under "What CI checks, and what it does not," and says the
full-stack walkthrough is a manual step run by whoever touches the directory. That is
an honest state, not a hidden gap, but it means:

• Nothing asserts that a run produces a trace that lands in Tempo. workflow.test.yaml
  tests workflow.yaml against flow test's stub engine (ran: [announce, settle, rollout, report]) — that is a workflow-semantics test with zero telemetry involved,
  not a pipeline test. It gives no signal about the collector, Tempo, Loki, or
  Prometheus.
• Nothing asserts a log line reaches Loki, or a metric reaches Prometheus, in CI or
  otherwise. The dashboard JSON (flowstate.json) references Prometheus series like
  rpc_server_duration_milliseconds_count, temporal_workflow_endtoend_latency_seconds_bucket,
  and a Loki stream selector {service_name=~"flowstate-.*"} — none of these names are
  checked against what the binary actually emits by anything that runs. The README's own
  panel 8 ("Everything the SDK is actually emitting") exists specifically because metric
  names drift through three components that may each append a unit or suffix, and the
  only way to notice today is a human staring at that table.
• #401 already recorded the upstream half of this gap: telemetry testing covers logs
  over the wire and traces in a recorder, and metrics not at all in-process. This issue
  is the downstream half — even a correctly-emitted metric is unverified once it has to
  survive the collector's Prometheus exporter and land in a panel query.

So: a compose file that parses is not a working stack, and today nothing in this repo
checks that it is one beyond a human running the walkthrough.

3. The ClickHouse golden path — what #423 already covers, and what's still open here

#423 scopes the stack (examples/observability-clickhouse/), the schema design
(tuned ORDER BY, materialized columns off the attribute registry, TTL retention),
and the moby-SDK integration tier that starts a real ClickHouse container and asserts
against it. That is the right shape and this issue does not re-litigate it. Two things
worth stating plainly here because they bound the decision:

What's gained: one store instead of three (Tempo + Loki + Prometheus), one query
language (SQL) instead of three (TraceQL, LogQL, PromQL), one retention mechanism
(TTL) instead of three, and it composes with correlation for free — trace id, span id,
and log records in the same engine mean a join is a WHERE, not a cross-datasource
link config like datasources.yaml's tracesToLogsV2.

What's lost or risky, checked against the exporter's own README
(opentelemetry-collector-contrib/exporter/clickhouseexporter) rather than assumed:

• Stability is uneven per signal — traces and logs are "beta," metrics are "alpha,"
  profiles are "development" and need ClickHouse 26.2+ and a feature gate. #423's own
  open question ("verify, don't assume" on whether Temporal SDK metrics land cleanly)
  is not paranoia; alpha is the exporter's own word for that signal.
• The exporter can auto-create its schema, but its README recommends
  create_schema: false in anything that isn't a lab, to avoid DDL races on
  deployment — which means the golden path owns migrations, not the exporter. #423's
  "schema is designed, not defaulted" section already commits to this; it is the
  correct call and it is also the ongoing maintenance cost of the golden path.
• Grafana's ClickHouse datasource claims first-class trace/log/metric support, but its
  own docs do not publish the column contract those panels expect beyond "pre-built
  dashboards expect tables named otel_logs and otel_traces." That is a real
  constraint on #423's "materialized columns lifting flowstate attributes out of the
  map" plan: whatever shape the schema lands in, the panels have to be built and
  checked against actual Grafana rendering, not assumed compatible because the plugin
  advertises the signal.
• High-cardinality attributes stored in a map column (rather than the sort key) are
  the exporter's own documented weak point for filtered queries — directly relevant to
  invariant 3 (cardinality is a bound the peer often controls): a workflow name or
  trigger id used as a filter predicate against an unindexed map column is a slow
  query under load, not just a slow query in the lab.

None of this is a reason not to build #423. It is the honest cost column next to the
gains, and the tripwires #423 already lists (JSON vs Map, profiles) are the right
place to track the first two.

4. Dashboards that are verified, mechanically

This is the piece invariant 6 asks for and #423 references but does not spell out
end to end: what does "a dashboard we validated" mean as a check that runs, not a
claim.

Proposed mechanism, independent of which backend a dashboard targets:

1. The panels reference names from one place. Every metric name, log field, and
   span attribute a dashboard's targets[].expr (or ClickHouse SQL) touches has to
   come from a registry, not be typed by hand into JSON. Invariant 1 already requires
   one attribute schema shared by spans, logs, and metric labels; #422 is where that
   registry is scoped to land. Once it exists, grafana/dashboards/flowstate.json
   stops being hand-typed strings like rpc_server_duration_milliseconds_count and
   service_name and instead is built or checked against the same constants the
   emitting code uses.
2. A test walks the dashboard JSON and extracts every identifier a query touches —
   metric names in a PromQL expr, label names in by (...) and =~"..." matchers,
   the Loki stream selector's label, the derived-field matcher in
   datasources.yaml (trace_id) — and asserts each one is a name the registry
   actually defines, or one of a small allowed set of infrastructure labels Prometheus
   and the collector add on their own (job, instance, and the like, which are not
   flowstate's to own). This is static: it does not need a running stack, so it costs
   nothing more than a Go test over a JSON file plus the registry.
3. A second, heavier check confirms the names are not just declared but actually
   emitted. This is the shared-case-driven part: run a workflow through the manual
   metric reader #401 is scoping for in-process metrics, and through the trace
   recorder already in use for span tests, and assert that every name the dashboard
   references appears in what was actually produced by a real run — not just that the
   registry has an entry for it. A renamed attribute that the registry still lists
   stale would pass check (2) and fail this one, which is the case that actually
   matters: the panel goes silently empty in production while every static check
   stays green.
4. For the ClickHouse path specifically, #423's integration tier is where check (3)
   naturally extends: once a real run's spans, logs, and metrics are queryable over
   SQL in a live container, asserting the dashboard's column and table references
   resolve is the same query the panel would run, executed by the test instead of by
   a person opening Grafana.

Check (1) ties dashboards to the attribute schema the way invariant 1 asks for. Check
(3)/(4) is what makes "verified" mean something beyond "the JSON is well-formed" —
which is exactly the gap docker compose config -q leaves for the compose file
itself, one layer up.

5. Where each piece belongs in CI

Not proposing a full stack per PR — a docker compose up of eight services (six
pulled images plus a Temporal cluster) per PR is slow and gives a false-negative
surface (an upstream image tag going stale) with nothing to do with the diff. Split
by cost and by what a failure would mean:

• PR CI (fast, every push): docker compose config -q for both the existing
  stack and any ClickHouse sibling stack (already true for the former, proposed for
  the latter per #423 §1). Dashboard-JSON check (1)/(2) above — pure static analysis
  over a JSON file and a registry, no containers, belongs here too; it is exactly the
  shape of the existing flow docs generate / reference-mirror drift checks CLAUDE.md
  already describes, and it should fail a PR the same way a docs-drift check does.
• Weekly deep tier (deep.yml): the full emit-to-panel round trip. Standing up
  the collector, Tempo/Loki/Prometheus (or ClickHouse, per #423's moby-SDK tier),
  running a shared-case workflow, and asserting check (3)/(4) above — every name a
  dashboard touches was actually emitted, at rest, in a live store — is exactly the
  shape deep.yml's existing legs already take (soak, examples-matrix, schedule
  search): expensive, real containers, files a deep-tier-labeled issue on failure
  rather than blocking a PR. This is also where #423's moby-SDK integration tier
  belongs outright, per its own house-gate language.
• Manual walkthrough: stays manual for the parts that are inherently visual —
  does the panel look right, is the layout sane — the README's existing walkthrough
  already covers this and should keep doing so. The point of (3)/(4) landing in the
  deep tier is to remove "does the data even arrive" from that human's job, not "does
  it look good."

Recommendation

1. File the dashboard-verification mechanism (section 4, checks 1-2) as its own
   near-term piece, gated on #422's attribute registry landing — it is cheap, static,
   and closes the loudest part of invariant 6 without needing a running stack or a
   ClickHouse decision at all.
2. Let #423 carry the ClickHouse golden path and its integration tier as scoped; this
   issue's section 3 is the trade-off column to weigh before committing engineering
   time to it, not a reason to delay #423's static-check half.
3. The heavier check (section 4, checks 3-4) rides on top of whichever backend lands
   first in the weekly deep tier — it does not require ClickHouse specifically, so it
   should not block on #423 landing.
4. Recommend the existing Tempo/Loki/Prometheus stack stays the default path and the
   ClickHouse stack stays a sibling, exactly as #423 already frames it — the evidence
   above (uneven exporter maturity per signal, undocumented Grafana column contract,
   map-column cardinality risk) is real but not disqualifying; it is the reason to gate
   ClickHouse adoption on #423's own tripwires rather than on this issue.

Start by reading examples/observability/docker-compose.yaml, grafana/dashboards/flowstate.json, otel-collector/config.yaml, workflow.yaml, and workflow.test.yaml, then review dependencies #422 and #423. The work is done when the agreed dashboard checks and emit-to-panel verification are implemented in the appropriate PR or deep CI tier without replacing the existing manual visual walkthrough.