GoogleCloudPlatform / GoogleCloudPlatform/scion

telemetry: clarify authoritative work-completion signal for external subscribers

Open
#161 1 comment 0 reactions 0 assignees View on GitHub
area:telemetry type:spike
Dominant language
Go
Stars
1.7k
Forks
263
Avg merge
7h 6m
Merged PRs (30d)
447

Description

## Summary

Scion's telemetry surface today doesn't offer an obvious subscription point for consumers that need to reconcile "what the plan said should happen" against "what actually happened." Before anyone can write a `TELEMETRY.md`, there's a design question worth settling: **what is the intended authoritative channel for "work done by an agent" that external systems should subscribe to?** Once that's answered, documentation and any behavioral alignment follow.

This issue is scoped to the design discussion. Concrete docs and fixes can spin out as separate issues/PRs once direction is clear.

## Context

We're building an external reconciler for an autonomous agent framework that dispatches N agents against a plan and needs a durable "what actually ran" signal to compare against. OTel looked like the right substrate, but a source read on current `main` (commit `22dfe46a`) surfaced three mismatches between what the surface *looks like* and what it actually emits.

## Findings

All OTel span emission lives in one file: `pkg/sciontool/hooks/handlers/telemetry.go`. Tracer instrumentation scope: `github.com/GoogleCloudPlatform/scion/pkg/sciontool/hooks/handlers`. No OTel imports in `pkg/agent/`, `pkg/harness/`, or `pkg/runtime/` — tracing is entirely a sciontool CLI concern.

Spans emitted (13 total):

- **Agent lifecycle** (5): `agent.lifecycle.pre_start`, `agent.lifecycle.post_start`, `agent.lifecycle.pre_stop`, `agent.session.start`, `agent.session.end`
- **Agent turn & prompt** (5): `agent.turn.start`, `agent.turn.end`, `agent.user.prompt`, `agent.response.complete`, `agent.notification`
- **Tool calls** (2): `agent.tool.call`, `agent.tool.result`
- **GenAI API** (2): `gen_ai.api.request`, `gen_ai.api.response`

### Three mismatches

**1. `sciontool status task_completed` emits no OTel spans.**

The `status` command (`cmd/sciontool/commands/status.go`) calls `StatusHandler.UpdateActivity()`, `LoggingHandler.LogEvent()`, and `hubClient.UpdateStatus()` — no `TelemetryHandler` is created or invoked. The same is true for `sciontool hook task_completed` (`runTaskCompleted()`). The nearest OTel equivalent, `agent.session.end`, only fires when the harness emits a `session-end` hook via `sciontool hook --dialect=claude`, which is a different lifecycle event with different semantics.

So an OTel-subscribing consumer looking for "task done" signals doesn't see the event from the `status` command — it sees session boundaries from the harness instead.

**2. Paired spans don't actually pair in the default deployment.**

`agent.turn.*`, `agent.tool.*`, and `gen_ai.api.*` are implemented as start/end pairs via an `inProgressSpan` `sync.Map` (`telemetry.go:232` start; `telemetry.go:261` end). But that map only spans calls inside the long-lived `sciontool init` process. In hook-per-process mode — which is the normal deployment — the end call can't find the start and falls back to `singleSpan` at `telemetry.go:249-254`. The fallback is documented in a code comment as "common in hook-per-process mode."

Net effect: for most deployments, the paired spans are two disjoint single-shot spans with `duration_ms` computed per-event, not a real span pair. Consumers relying on span parent/child relationships or OTel's native duration won't get what the span names imply.

**3. `agent.user.prompt` is silently dropped by default.**

`SCION_TELEMETRY_FILTER_EXCLUDE` defaults to `["agent.user.prompt"]` (see `filter.go`). Any consumer subscribing to it gets nothing — no log, no warning. Reasonable privacy default, but worth flagging because it means schema-critical spans can disappear by default configuration.

### Supporting findings (less load-bearing, listed for completeness)

- Resource attributes are dot-namespaced (`scion.grove.id`, `scion.harness`); metric labels on the same values are underscored (`agent_id`, `grove_id`). Pick one.
- No sampler is configured on any `TracerProvider` — defaults to OTel SDK's `AlwaysOn`. Fine for small volumes, worth a sampling story for the Hub deployment case.
- No stability markers anywhere — no `// EXPERIMENTAL`, no `v1`, no versioned attribute keys. Consumers have no signal about what's safe to depend on.

## Design questions for maintainers

1. **What's the canonical "work done by an agent" signal Scion wants external systems to subscribe to?**
- Hub status API (poll-based, fed by `sciontool status`)?
- OTel spans emitted by sciontool hooks (push-based)?
- Both, with clear division of responsibility?
- Something else in-flight?

2. **If OTel is part of the answer, should `sciontool status` emit spans** (so the command name and the telemetry surface agree), or should the Hub status API be documented as the authoritative work-completion channel and OTel scoped to lower-level operational telemetry?

3. **For paired spans in hook-per-process mode** — is the intent to make pairing work across processes (e.g., persist `inProgressSpan` state to disk or the Hub), or to redesign these as single-shot spans with `duration_ms` as a first-class attribute (which is what they effectively are today)?

4. **Filter defaults** — is silent drop intentional? If so, worth documenting prominently. A debug-log on filter hit would make this discoverable.

5. **Stability contract** — do the current span names and attribute keys count as public API, or still experimental? Once that's declared, the documentation issue becomes straightforward.

## What a good resolution looks like

Not a spec, just alignment: a short comment on this issue from maintainers picking a direction on #1, with enough context that a follow-up `TELEMETRY.md` PR and any behavioral-alignment issues can be scoped narrowly. Happy to do the documentation PR legwork once direction is clear.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pkg/sciontool/hooks/handlers/telemetry.go, cmd/sciontool/commands/status.go, and filter.go; compare the status and hook task-completion paths with the emitted OTel spans. Done means maintainers choose and record the canonical external work-completion signal, including the intended role of OTel and Hub status, so follow-up documentation and behavior issues have a narrow scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.