observability: sampling, sensitive-data containment, a stable attribute schema, and the profiles signal

cmd/flow/telemetry.go is a good foundation, honestly documented: all three OTLP exporters, the otelslog bridge, the Temporal interceptors and metrics handler, semconv v1.41, gated on the standard OTEL_EXPORTER_OTLP_ENDPOINT so a first run phones nowhere ("telemetry is a deployment's choice rather than a default phone-home"), with the flush, resource, and propagator lessons written down. examples/observability/ stands up the full collector/Tempo/Loki/Prometheus/Grafana path in CI. This issue is the next four pieces, each held to an existing house doctrine, and each checked against the OpenTelemetry guidance it corresponds to (sampling: opentelemetry.io/docs/languages/go/sampling; sensitive data: /docs/security/handling-sensitive-data; profiles: /docs/concepts/signals/profiles).

1. Sampling is currently undecided, which means always-on

No sampler is configured, so the SDK default (ParentBased(AlwaysSample)) applies. For a workflow engine that is the wrong silent default at any real volume: every step of every run is a span, and a busy worker's trace bill scales with throughput nobody chose. The fix is not a flowstate opinion — it is wiring the standard knobs and saying so: the Go SDK honors OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG (parentbased_traceidratio etc.) when the provider is built with the env-aware sampler; use that path, document the variables in the hand-kept env-var table (cmd/flow/docsgen.go — the docs-drift check then keeps it honest), and demonstrate a tail-sampling collector config in examples/observability/ for the keep-all-errors posture head sampling cannot express. Deployment decides; the file never speaks (report-file-properties rule, applied to telemetry). One flowstate-specific decision worth making deliberately: errored runs and compensation paths are the traces someone will actually need — the example config should show tail-sampling keeping 100% of error traces while sampling successes, because that is the configuration every operator wants and few find on their own.

2. Sensitive data: complete the containment invariant across signals and layers

The in-process half of this is further along than a reader of the secrets doctrine might guess, and #401 has already mapped it precisely: logs are asserted end-to-end over OTLP including secret redaction (cmd/flow/telemetry_test.go:694-855, a real httptest collector decoding gzipped protobufs), traces have requireNoSecretInSpans over a span recorder (engine/tracing_test.go), and metrics have nothing — no reader, no metricdata assertion anywhere in the tree. #401 owns closing that third surface (its item 1 — metrics attributes deserve the same no-secrets property, completing the invariant across all three signals); this issue doesn't restate that work, it builds the two layers beyond it, per the OTel sensitive-data guidance:

• Collector-side (defense in depth, documented not owned): the example pipeline gains the contrib redaction/attribute processors with a config that names the attribute keys flowstate emits — which requires §3's schema — so an operator inherits a scrubbing posture rather than designing one. Documented as second layer, explicitly: the in-process tests (#401's completed set) are the guarantee; the collector is the backstop for operator-added instrumentation.
• At rest: #423 extends the same property to storage — asserting over SQL that no secret material survives into any signal table. Emission-clean and storage-clean are separate claims (a processor added between them can regress either), so both get tested.

3. The attribute schema is a contract, and today it is implicit

Dashboards, tail-sampling policies, and redaction configs all key on attribute names — flowstate.workflow.name, flowstate.step.id, whatever the engine emits today by accident of implementation. That set is exactly the kind of surface the docs-drift machinery exists for: define the attribute names as constants in one place, document them as a generated reference page (the diagnostics-codes precedent: registry + generated doc + a test that nothing emitted is undocumented), and version changes to them as breaking, because renaming an attribute breaks every dashboard and sampling policy in the wild — buf breaking logic, applied to telemetry. Follow semconv where a convention exists (upgrade cadence for the pinned semconv/v1.41.0 import gets decided here too) and mint flowstate.* names only where none does.

4. Profiles: the fourth signal, tracked deliberately, adopted when stable

OTel profiles are the newest signal (OTEP 239 lineage; collector support maturing, Go SDK support early). For flowstate the payoff is worker-side: a continuous-profiling pipeline correlated by the same service.name/resource so an operator pivots trace → profile when a step is slow for CPU reasons rather than wait reasons. The honest posture today: do not build against an unstable SDK surface; do prepare the two things that cost nothing and make adoption free later — resource identity already consistent (it is: one newResource feeds all providers — keep it that way when profiles arrive), and the examples/observability collector pinned at a version whose profiles support can be enabled behind a compose profile flag when it stabilizes. Revisit on a stated trigger: profiles marked stable in opentelemetry-go, or the collector's profiles pipeline GA, whichever lands first — a tripwire, not a date.

Where the data lands

This issue deliberately stops at emission: flowstate speaks OTLP and has no storage opinion. #423 builds the storage golden path — a ClickHouse-backed variant of examples/observability/ with a designed schema, TTL retention, Grafana datasource, and container-backed integration tests — consuming the attribute registry this issue defines.

House gate

Sampling env-vars wired and in the generated env-var table; tail-sampling example config in examples/observability/ with the keep-errors posture, docker compose config -q still green in CI; the collector redaction config landed and keyed to the attribute registry (#401 gates the third in-process containment surface — metrics — and this issue's gate includes that issue being closed, so the invariant is whole before the registry is called done); the attribute registry + generated reference page + nothing-undocumented test; and the profiles tripwire recorded in this issue rather than in anyone's memory.

Open

• Whether Temporal SDK metrics (task-queue depth, workflow-task latency) get exemplar/attribute alignment with the flowstate attribute schema or stay upstream-named (probably upstream-named — renaming other people's metrics is its own sin).
• Whether log sampling (the newer OTEL_LOGS_* knobs / log record processors) is worth wiring now or waits for a real volume report — logs currently ride the activity path, which is already bounded.
• Span-per-step granularity itself: whether a for_each over 10k items emits 10k child spans (bound it — the same "outside party grows it" rule; likely a span-per-iteration cap with a summarizing event beyond it, decided in this issue not discovered in an invoice).

Start with cmd/flow/telemetry.go and cmd/flow/docsgen.go, then read the existing checks in cmd/flow/telemetry_test.go and engine/tracing_test.go and inspect examples/observability/. Run docker compose config -q before changes. Done means the sampling variables and generated table, collector redaction configuration, attribute registry and generated reference with coverage test, and profiles tripwire are documented and validated, with #401's metrics containment complete.