observability workstream: log/trace correlation, one attribute schema, and sensitive-data containment across the telemetry pipeline

Workstream of #522 (invariants 1, 2, and 6), and the "logs" half of the containment work #422 scopes in its section 2. This is a research pass only, no code changed. Every claim below is either a file:line citation or flagged as unsure with the specific path named, per invariant 6 ("verified, not demonstrated").

1. Trace/span correlation: real on one path, absent on three

Correlation is real, but only where the record is emitted with a context that carries a span, and the repo's own doc comment is explicit about the boundary (cmd/flow/telemetry.go:262-289):

• A log: step running through Temporal correlates. taskFuncLog calls LoggerFrom(ctx).LogAttrs(ctx, …) (pkg/flowstate/v1/eval_task_library.go:181-182), and withActivityLogger fans that record to both Temporal's own logger and otelslog.NewHandler (pkg/flowstate/v1/engine/activitylog.go:48-53). The activity's context carries the span Temporal's tracing interceptor opened, so the bridge stamps trace/span id on the record.
• A log: step in flow run local does not correlate — no RPC, no span, so the record exports with no trace id (cmd/flow/telemetry.go:265-269).
• The server's and worker's own infra lines do not correlate. infraLogger() (cmd/flow/main.go:176-177) logs through Info/Warn, not the Context variants, at moments outside any request span (cmd/flow/telemetry.go:274-277).
• Webhook receiver lines (pkg/flowstate/v1/server/webhook.go:496-596, e.g. r.log.WarnContext(req.Context(), "refused a delivery: …")) do use the Context variant and reach the OTLP-bridged infra logger built in main.go. Whether the incoming HTTP request actually carries a span at that point is not established anywhere I read — nothing in webhook.go or its caller in cmd/flow/webhooks.go starts or extracts a span for a delivery request. I could not confirm correlation here either way; it needs a test the way TestConfiguredSlogCallReachesTheCollector (cmd/flow/telemetry_test.go) tests the activity path.

Test coverage matches the claim: cmd/flow/telemetry_test.go asserts the local and activity cases. Nothing asserts (or denies) correlation for infra lines or webhook lines.

2. One attribute schema: not yet one schema

Invariant 1 says a run id, workflow, step, tenant, and trigger must be spelled the same way in a span attribute, a log field, and a metric label. Today they are spelled differently depending on which signal you look at, and some are missing entirely from spans:

• Span attributes minted in startTaskSpan (pkg/flowstate/v1/engine/activities.go:308-338): flowstate.task.name, flowstate.step.id, flowstate.attempt, flowstate.secret.refs, flowstate.secret.ref.count. No run id, no workflow name, no tenant, no trigger attribute anywhere in this file, or anywhere else I found under pkg/flowstate/v1/engine.
• The log: task's own record (pkg/flowstate/v1/eval_task_library.go:176-182) carries only the message and the author's fields: map — no run/workflow/step/tenant/trigger attribute is attached at the emission site. Temporal's own activity logger separately tags workflow id, run id, activity type, and attempt (pkg/flowstate/v1/engine/activitylog.go:18-21, 34-38), but that tagging happens only on the branch that goes to Temporal's logger, using Temporal's own field names, not flowstate.*. It is a different, differently-spelled copy of roughly the same facts, not the same schema shared across signals.
• Webhook logs use bare "workflow", "webhook", "delivery", "run" as slog keys (pkg/flowstate/v1/server/webhook.go:530-531, 561-562, 578-579, 592-594) — a third spelling again, unrelated to flowstate.task.name/flowstate.step.id.
• Plugin spans and metrics use their own set: flowstate.plugin.name, flowstate.plugin.operation, flowstate.plugin.outcome, flowstate.task.name (pkg/flowstate/v1/plugin/telemetry.go:40-53).

So today there is no single flowstate.run.id (or equivalent) that a span, a log record, and a metric label all carry, and #522's invariant 1 is unmet as stated. This is exactly the registry work #422 section 3 already scopes; this issue is the evidence for why it also blocks correlation, not just dashboards.

3. Containment audit: the important half

CLAUDE.md's rule ("secrets never enter workflow history") and #522 invariant 2 extend it to spans, logs, metric labels, and exemplars. checkSensitiveLog and secretReferenceAttributes show the repo already thinks this way for the paths it covers. What I could verify, and what I could not:

Confirmed safe, with a stated boundary:

• startTaskSpan never writes an input, output, or error message onto a span — only the task name, step id, attempt, and secret references (never values), documented in pkg/flowstate/v1/engine/activities.go:282-292.
• recordTaskOutcome (pkg/flowstate/v1/engine/activities.go:372-389) deliberately records only the error's classification, never RecordError, "since RecordError writes the message into" a span event, and a task's error message can quote what it was given (http URL, plugin's own text).

A gap between the stated rule and the code that should follow it:

• pkg/flowstate/v1/plugin/telemetry.go:60 calls span.RecordError(err) on every failed plugin operation. This is the exact thing activities.go:288-292 names as unsafe two directories over: a plugin's error can quote whatever the plugin process wrote back, and RecordError puts that message into a span exception event verbatim. I did not find a redaction step between a plugin's returned error and this call. This looks like the same class of leak the doctrine already names, just not yet applied to the plugin package.

Named, not resolved — paths I could not confirm are safe or unsafe:

• log: step fields: map. sensitive_log.go:57-61 says outright this sink is not checked: ${inputs.token} inside a fields: value compiles to a map-building expression, not a bare reference or + chain, so the lint's direct-surfacing rule does not see it. A sensitive input placed in fields: instead of message reaches the log, and now also OTLP, uncaught.
• Derived-but-still-identifying values in message. The lint explicitly treats anything wrapped in a call as derived and does not report it (sensitive_log.go:42-48), including ${string(inputs.token)} as a whole message with no concatenation, and any slicing/hashing that leaves enough of the value recognizable (a 4-char prefix, a low-entropy hash). This is a stated, accepted gap, not a bug, but it is still a containment gap once the sink is OTLP rather than only run history.
• HTTP task error messages. activities.go:287-288 says "an http task's error names the URL it called" — the URL itself may carry credentials or tokens in a query string, and the same comment block already refuses query-string secrets as a destination (eval_task_library.go:133-140) for a related reason. Whether that error message, once returned to the workflow, can itself end up in a place telemetry reads (a span event via some other path, a log line) I did not fully trace.
• Webhook delivery bodies. decodeDeliveryBody (pkg/flowstate/v1/server/webhook.go:808-816) wraps a JSON decode failure with %w, which for Go's json.Decoder is a position/type error, not a value dump — likely safe, but I did not verify this against every error shape the decoder can produce for malformed input.
• Plugin arguments. I did not find where a plugin's launch or call arguments are (or are not) written to a span or log line; pkg/flowstate/v1/plugin/plugin.go and telemetry.go need a closer read for this than this pass gave them.
• CEL evaluation errors. pkg/flowstate/v1/eval.go has many fmt.Errorf sites that echo type names, step ids, and field names, which look like classifications rather than values (e.g. eval.go:850, 941) — but I did not check every one against whether it can carry an evaluated CEL value into the message, only spot-checked a sample.

4. The log: task output over OTel, and the lint's blind spot

Once a log: step's OTLP bridge is active, the message and every fields: entry are exported as log record attributes with no further filtering (activitylog.go fans the record straight through; nothing between LogAttrs and the OTLP exporter redacts). checkSensitiveLog only inspects the message input's CEL expression for a direct reference to a sensitive: input (sensitive_log.go:70-81, 143-173), so:

• a derived-but-still-sensitive value in message (${string(inputs.token)} standalone, a 4-character prefix, a low-entropy hash) is not flagged and now travels to a collector, not just run history
• anything in fields: is entirely outside the lint's scope, flagged and value, and now travels to a collector too

Both are named as known gaps in sensitive_log.go's own doc comment, written for the run-history sink. The OTLP sink raises the stakes on the same known gaps without the file saying so.

5. Sampling and volume under a large loop

No sampler is configured (cmd/flow/telemetry.go builds sdktrace.NewTracerProvider with no WithSampler, so the SDK default ParentBased(AlwaysSample) applies — matches #422 section 1's framing exactly). startTaskSpan is called once per activity execution (activities.go:110, 162, 196), and a for_each/loop body's task runs as one activity per iteration with no iteration cap on this path — I found no span-count bound anywhere between the loop executor and startTaskSpan. A workflow that loops thousands of times therefore opens thousands of spans, unsampled, each carrying flowstate.secret.ref.count and a flowstate.secret.refs string slice per iteration if the body task uses a secret. This is the exact open question #422 already records under "Open" (span-per-iteration, "likely a span-per-iteration cap with a summarizing event beyond it, decided in this issue not discovered in an invoice") — this pass did not find that decision made anywhere in the tree, only the open question.

How containment should be tested

Following CLAUDE.md's "secrets never enter workflow history" section, applied to telemetry rather than to error values — the shapes that matter are the ones a value can hide behind, not just the value printed bare:

• %v, %+v, %#v, and %s on: a span's recorded attributes, a metricdata export, and a log.Record/OTLP log entry — matching the existing shape of requireNoSecretInSpans (engine/tracing_test.go) but run against all four printf verbs, not just an equality check on attribute values.
• The same four verbs on a struct that holds one of the above (a batch of spans, a plugin telemetry snapshot) — reflection through an unexported field is the leak class CLAUDE.md already names, and it applies identically whether the struct is being logged or being handed to a redacting exporter.
• The same four verbs on a slice of spans / log records / metric points — a leak that only shows up at index 2 of a batch is invisible to a test that only constructs one record.
• A real collector test per signal, the way cmd/flow/telemetry_test.go:694-855 already does for logs over OTLP (gzipped protobuf decode against an httptest server) — #401 already tracks doing this for metrics; this issue's ask is that once that lands, the same harness gets a sensitive: input through a log: step's fields: map and a plugin RecordError path, both flagged above as currently uncovered, and asserts neither reaches the wire.
• A negative-direction test for the lint itself, in the shape CLAUDE.md's "test that A cannot reach B" section asks for: not just "a bare ${inputs.token} in message is flagged" (already true), but "a sensitive: input placed in fields: is not silently accepted as safe" — i.e., a test that currently must fail, recording the known gap as a red test rather than a comment, until fields: is in scope.

Suggested scope for whoever picks this up

1. Close the RecordError gap in pkg/flowstate/v1/plugin/telemetry.go:60 to match the classification-only rule activities.go already states.
2. Extend sensitive_log.go to cover log:'s fields: map (the file already names this as the obvious next scope).
3. Land the run id / workflow / step / tenant / trigger attribute set as the shared vocabulary #422 section 3 scopes, and use it in activitylog.go's emitted record so a log: step's OTLP-exported line and its span agree on spelling, not just on trace id.
4. Decide and implement the span-per-iteration bound #422 already flags as open.
5. Write the collector-backed containment tests described above, extending the harness cmd/flow/telemetry_test.go already has for logs.

Start by reading #422 and #522, then inspect cmd/flow/telemetry.go and telemetry_test.go for existing correlation and OTLP coverage. Trace the named paths in pkg/flowstate/v1/engine, pkg/flowstate/v1/plugin, pkg/flowstate/v1/server, and sensitive_log.go. Done means the open schema, containment, sampling, and webhook-correlation questions have decisions and the proposed collector and lint tests cover the identified gaps.