Add run-level observability and BI telemetry for IssueLens

Problem and desired outcome

IssueLens needs two complementary views:

• Run diagnostics: Explain a particular invocation or chat turn: which agents and tools ran, which model calls consumed tokens, where time was spent, and what failed.
• BI reporting: Track usage, performance, reliability, and verified work across repositories, issues, and pull requests over time.

Use OpenTelemetry with Foundry's supported Application Insights integration rather than creating a second observability platform. Keep a small, versioned, unsampled operational dataset for BI separate from optionally sampled diagnostic spans. Do not derive business counts from displayed output or sampled traces.

Scope: Both Foundry hosted-agent protocols, all IssueLens agent roles, the bundled GitHub MCP server, and in-process tools. This is a source review and proposed design, not a report of production measurements. No live Azure telemetry was queried and no deployment is authorized by this proposal.

Current baseline

Reviewed repository revision: 49df3d97547069f891a68248be6ed722c2aeca2f.

Surface	What exists	Gap this request addresses
Host and client	One IssueLensHost and a lazily started, shared CopilotClient; no explicit Copilot telemetry configuration or application-owned telemetry adapter in the reviewed source.	Deliberate exporter configuration, cross-process correlation, schema, and privacy controls. Platform-provided traces must be inspected before adding instrumentation; their production coverage is not established by this review.
Invocations	A new Copilot session per request; SDK events forwarded as SSE.	Events are not normalized into persistent run, model, tool, target, or outcome records.
Responses	A session resumed per conversation; the callback handles assistant deltas, idle, and session errors.	Shared telemetry collection must also observe usage, tool, subagent, retry, and failure events without changing the response protocol.
Actions	The stream renderer already shows elapsed time, observed tool calls, failed tools, retries, and subagent activity.	This is bounded presentation, not authoritative BI storage. Display limits and publication settings must not affect measurement.
GitHub tools	Explicit repository and typed issue/PR arguments, validated GitHub operations, and tool-confirmed results.	These are useful measurement boundaries for targets and actual writes; prompt mentions are not reliable work counts.
Content logging	Chat currently logs textual input; media redaction removes inline binary data, not all user content.	Adding an exporter must not inadvertently turn current content logs into persistent sensitive telemetry.

Source: client/session wiring, invocations, chat, Actions presentation, GitHub request boundary.

Verified platform support

Public documentation and source were reviewed for Copilot SDK v1.0.7, the repository's declared minimum, and the Azure source revision associated with Invocations 1.0.0b7 and Responses 1.0.0b9. These establish available integration points, not the versions or behavior of the deployed agent. Both Azure packages have an open-ended core dependency, so their pins do not pin the whole telemetry stack.

Verified capability or constraint	Consequence for IssueLens
Copilot SDK native OTel provides invoke_agent, chat, and execute_tool diagnostics. Python TelemetryConfig supports otlp_endpoint, otlp_protocol, exporter_type, file_path, source_name, and capture_content.	No framework migration is needed. Native telemetry is useful but does not replace business accounting. Explicitly choose HTTP/protobuf when using the proposed OTLP route.
SDK create/resume/send inject the active W3C context per RPC. Ordinary event callbacks do not restore that context; the bridge propagates traceparent/tracestate, not arbitrary baggage.	Retain the singleton client. Bind each turn's observer explicitly; do not expect business metadata to cross processes automatically.
Hosted-agent telemetry uses a project-provided, reserved APPLICATIONINSIGHTS_CONNECTION_STRING and runtime-managed export.	Reuse the existing Python pipeline; do not override that variable in agent.yaml or initialize a competing global provider. Project monitoring still needs to be configured and verified.
The inspected host bootstrap defaults its content-capture setting to true when the environment setting is absent. Its span enrichment can also overwrite standard agent identity with hosted-agent identity.	Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false before host construction and Copilot capture_content=False. Preserve logical role separately, for example issuelens.agent.role.
Native CLI export is separate from the Python exporter. Azure documents a collector-mediated, authenticated OTLP ingestion route, not a Copilot configuration that takes an Application Insights connection string as an OTLP endpoint.	Validate the receiver, authentication, identity enrichment, tables, and Foundry drill-down before enabling native export. Do not embed an ad hoc OTLP bridge in IssueLens.

Important: Setting a shared OTEL_EXPORTER_OTLP_ENDPOINT can affect both host and CLI export, potentially creating duplicate telemetry. Python-provider flush does not prove CLI export has flushed. Treat exporter routing, sampling, and shutdown behavior as explicit integration work.

Proposed architecture

1. Define the accounting unit

One run is one accepted /invocations request or one /responses turn, not an entire conversation or Copilot session. A retry inside that run is a separate attempt, not a new business task. A caller resubmission is a new run, linked to the same source operation where verified identifiers exist.

Create an opaque run_id at the host boundary and retain the applicable invocation ID, response ID, conversation ID, Copilot session ID, trace ID, deployment version, source commit, SDK/runtime versions, and telemetry schema version. Conversation and session identifiers are correlation fields, not metric dimensions.

Represent malformed requests as admission failures separately from accepted runs. Keep three independent fields: transport_status, execution_status, and business_outcome. Successful streaming is not proof of successful triage, planning, notification delivery, or wiki publication. Use explicit outcomes such as confirmed action, no-action, denied, partial, and unknown; generic free-form agent output does not have to become JSON merely to support telemetry.

2. Instrument real execution, not model narration

Use one request-bound telemetry adapter for both host handlers. It subscribes before sending and before protocol presentation filters, observes events without consuming or altering them, owns bounded state, and remains active until streaming completes, fails, or is cancelled. Late events must not be assigned to the next resumed turn.

Recommended first delivery: Send event-derived accounting and supported diagnostic spans through the existing Python hosting telemetry pipeline. This supplies useful BI without making new collector infrastructure a prerequisite. Prefer native Copilot model/tool/subagent spans once their separate export and correlation path is verified; at that point disable overlapping event-derived spans while retaining the accounting observer. Do not manufacture unavailable model or subagent attribution.

Use the SDK event contract deliberately:

• assistant.usage is per API call and ephemeral, so it must be collected live. Optional fields include input/output/cache/reasoning tokens and TTFT; Python durations are timedelta values. Provider/API support must be measured, not inferred from the schema. Deduplicate using scoped event/call identities where available, not timestamps or tool names.
• session.usage_info reports context-window utilization, not a billable token ledger. Keep it as an optional context-pressure metric. Aggregate AI-unit checkpoints are not substitutes for per-call token records.
• tool.execution_start/complete, subagent.started/completed/failed, model retries/failures, and session errors provide lifecycle observations. Prefer explicit agent/tool relationships. Event parentId is a previous-event link, not an agent-parent relationship; parentToolCallId is deprecated in relevant SDK schemas.

Enabling telemetry captures future runs; missing historical usage and execution events cannot be reconstructed from session history. Existing retained traces may provide a partial baseline only after their coverage is inspected.

The desired logical view is:

Hosted request / IssueLens run
  Host initialization, media loading, session creation or resume
  IssueLens orchestration
    Model call attempts
    Delegation / subagent execution
      Model call attempts
      MCP or in-process tool calls
        GitHub / notification / wiki dependency operations
  Terminal execution status and verified business observations

Treat the Python host, Copilot runtime subprocess, and stdio MCP subprocesses as separate instrumentation boundaries. A shared client must receive request-specific context; never set one startup-time trace parent for every future run. Callback context and correlation must be explicit, not a mutable global "current run."

Use standard gen_ai.* fields where supported and an issuelens.* namespace for product-specific fields. GenAI semantic conventions remain in Development; version the mapping. For example, current Copilot cache-write naming and the latest conventions differ. Select one authoritative measurement source per quantity so native spans, callbacks, HTTP auto-instrumentation, and run summaries do not count the same work twice. For token histograms, their sum measures tokens; their observation count measures calls.

3. Add a small BI contract

The following are proposed logical records, not existing tables or SDK event names:

Record	Grain and purpose
issuelens.run.started	One logical record per accepted run; enables detection of missing terminal records.
issuelens.run.completed	One logical terminal record per run, including failures and cancellations, with timing, token aggregates, observed work counts, outcome, and completeness fields. Deduplicate exporter retries by run_id and record identity.
issuelens.run.agent	One terminal summary per agent execution, with parent identity, exclusive usage when attributable, status, and duration. Keeps role-level BI available when detailed spans are sampled.
issuelens.run.target	One deduplicated association per run, target, and relationship, with target kind, canonical repository identity, number where applicable, and confirmed action counts.

Do not repeat whole-run token or cost totals on every target row and then sum the join. Multi-target runs need an explicitly documented allocation rule or an "unallocated/multi-target" bucket.

Keep these compact records outside diagnostic trace sampling, including ingestion-side sampling. Check trace-based log sampling as well: writing a summary log does not by itself guarantee retention when its trace is not sampled. They remain best-effort telemetry, not an exactly-once audit ledger: report dropped exports, missing completions, ingestion lag, and partial measurements. Do not silently turn an unfinished run into success.

Required measurements

Area	Required metrics and breakdowns	Accounting rule
Tokens and model calls	Input/output tokens; cache-read/cache-write and reasoning tokens when actually exposed; calls, attempts, retries, model/provider, per-run and per-agent usage.	Sum unique model-call attempts once. Preserve provider semantics: cache and reasoning breakdowns can already be included in totals. Do not add cumulative session usage again on each resumed turn. Missing usage is unknown, not zero.
Latency	Per-model-call TTFT, run time to first visible output, total run duration, model/tool/subagent durations, startup and session-resume time. p50/p95/p99 by protocol, job type, role, model, and version.	Keep model TTFT separate from user-perceived latency. Do not add overlapping child durations and label the result wall-clock time.
Tools	Started/completed/failed calls, duration, retries, tool name/type, agent ownership, read/write classification, dependency status class.	MCP failure results and in-process ToolResult failures count even when the transport succeeds. Distinguish a delegation tool from the agent execution it starts.
Subagents	Runs, successes/failures/cancellations, duration, nesting/delegation, fan-out, exclusive tokens, and attribution coverage for triage, find-criticals, plan, and team-memory.	Inclusive parent totals are useful for drill-down but must not be added to child totals. Unattributable usage stays in an explicit unknown bucket.
Errors	Validation/configuration/authentication, model/provider failures, throttling, tool/dependency failures, policy denial, timeout, cancellation, invalid expected output, and telemetry-export failure.	Use bounded error codes and stages. Separate recovered attempts, terminal failures, deliberate denials/no-action, and unknown outcomes. HTTP 200 and stream completion do not override execution failure.
Targets and adoption	Total runs, distinct conversations, distinct repositories, issues and PRs targeted/read/modified, activity over time, repeat runs per item, job-type distribution.	Count typed, canonical identities from validated execution evidence. A list/search result or textual mention is not proof that every returned issue was analyzed.
Confirmed actions	Successful label/assignment operations, comments/artifacts posted, notification submissions, wiki updates/no-change/conflicts, denied writes.	Count tool-confirmed operations; claim a state change only when the result or before/after evidence proves one. An idempotent label request is not necessarily a new label. A notification endpoint response confirms submission, not downstream delivery unless a receipt exists. Keep source project and wiki destination separate.
Efficiency and cost	Cache-use fraction, tokens/calls per completed run, costly/slow outliers, retries, context pressure/compaction where exposed, estimated model cost.	Show usage coverage. BYOK cost estimates require versioned pricing and model mapping; Copilot billing must not be inferred from token counts. Copilot's cost multiplier is not currency, and parent/child AI-unit totals must not be summed together. Billing systems remain authoritative.
Runtime and telemetry health	Active runs, cold starts, resume failures/fallbacks, process restarts, CPU/memory where the platform exposes them, dropped records, orphan spans, ingestion lag.	Keep platform health separate from business outcomes. Export failure must not trigger an agent retry or repeat a GitHub write.

TTFT definition

Publish separate measurements with explicit boundaries:

• Model TTFT: Record the SDK/provider-reported TTFT when available, preserving its definition. Native OTel time-to-first-chunk is a distinct measurement: the first chunk can be empty, reasoning, or tool-related rather than visible text. Neither is the duration of the entire tool loop.
• Run time to first root-agent output: Host request admission to the first nonempty, user-facing root-agent content emitted by the handler. Exclude SSE lifecycle messages, reasoning/analysis, tool events, and nested-agent output. Use message-start phase metadata when later deltas omit it.
• Time to final answer/completion: Request admission to final user-facing answer availability and terminal completion. This prevents a quick progress message from hiding a long run.

If useful, expose time to first activity separately. Where a channel intentionally displays nested-agent or progress output, measure that first-presented-output latency separately from root-answer latency; telemetry must not silently change presentation. A host measurement is not client end-to-end latency: workflow queues, Foundry ingress, network transit, and rendering require client-side timing. Missing first output, cancellations, and nonstreaming fallbacks need explicit states rather than zero-duration samples.

Target-counting definition

Use a repository ID when already independently resolved, plus a canonical repository name for display. Identify a work item by repository, kind (issue or pull_request), and number; a bare number is not unique. Do not make new GitHub reads solely to enrich telemetry.

Retain distinct relationships: requested, read as evidence, and modified. A related repository used for duplicate investigation is not automatically the primary task's repository. A wiki destination is not the source project's identity.

Authoritative workflow metadata and validated tool arguments/results can provide observations. Prompt text, model prose, and self-declared provenance cannot establish trusted attribution or business success. A telemetry-only correlation field grants no command or write authority. Preserve the current invocation contract unless an explicit backward-compatible extension is separately designed.

Exact distinct counts require the complete deduplicated target dataset. Sampling weights cannot reconstruct which unique issues were missing. If a query uses approximate distinct counting, label that fact.

Privacy, cardinality, and reliability requirements

No content capture by default. Exclude prompts, answers, reasoning, issue bodies, code, file/image contents, tool payloads, email addresses, credentials, and URL query strings. Allowlist the few necessary metadata fields instead of serializing complete events and attempting to redact afterward.

Remove or explicitly gate existing textual input logs before forwarding application logs to a persistent sink. Sanitize exception text and HTTP telemetry as well: notification URLs contain SAS signatures, and dependency paths can contain private repository information.

Use low-cardinality metric dimensions such as environment, protocol, supported job/agent/tool name, model, operation, outcome, and error category. Repository identifiers, issue/PR numbers, session/run IDs, and trace IDs belong in restricted records/spans, not histogram or counter labels.

Choose access controls and retention for the sensitivity of all observed repositories. Repository names and work-item numbers can themselves be confidential. Any future temporary content-debug mode requires explicit administrator enablement, short retention, and a separate review; it is not part of the MVP.

Reuse the host's existing telemetry provider when applicable; avoid duplicate global initialization and duplicate HTTP spans. Use bounded asynchronous export, bounded buffers, observable drop counters, and bounded shutdown flush. Telemetry must not block streams, change authorization, or influence model decisions. Exporter failures produce sanitized diagnostics, not silent loss or business retries.

Keep operational summaries unsampled. Diagnostic spans may be sampled after coverage is proven. Retaining every failed trace requires a supported tail-sampling design; ordinary head sampling cannot promise that after a trace was discarded.

User-facing views

Deliver an Azure Monitor Workbook first, with time/environment/version/protocol/job filters and four views:

1. Overview: Runs and outcomes, distinct active targets, usage, latency percentiles, cost estimates with coverage, and change over time.
2. Run explorer: Lookup by run, response/invocation, conversation, repository/work item, or workflow correlation; open the logical execution tree and correlated Foundry/Application Insights trace. Start with full diagnostic coverage in the small pilot; if sampling is later enabled, visibly identify runs whose detailed trace was not retained.
3. Reliability and performance: Slow/error runs, tool and subagent breakdowns, retries, throttling, startup/resume overhead, and release comparisons.
4. Telemetry quality: Missing usage/parents/completions, export drops, ingestion lag, sampling configuration, and measurement versions.

Workbooks can query Application Insights/Log Analytics, and Power BI can consume curated KQL results later if wider business reporting is needed. Do not make an additional BI service a prerequisite for useful diagnostics. Add configurable alerts for sustained error/timeout rates, p95 latency regression, unusual token/cost growth, and telemetry loss; establish thresholds from a baseline rather than inventing an SLO.

Acceptance criteria

• Both protocols use the same measurement definitions. Concurrent runs remain isolated, and two turns in a resumed conversation produce two run records without recounting historical usage.
• Create/resume/send under different active parents with the same singleton produce separate trace parents. Interleaved callbacks and late events cannot leak attribution across runs. If overlapping turns in one conversation are unsupported, enforce or explicitly reject that case rather than guessing ownership.
• A fixture with nested and parallel subagents produces correct ownership and deduplicated tool/model counts. Root, subagent, and model rollups reconcile without double counting.
• Usage tests cover cache/reasoning breakdowns, absent usage on failures, retries, cumulative session statistics, and unavailable per-agent attribution.
• TTFT tests exclude lifecycle, empty, analysis/reasoning, and nested-agent events; distinguish model TTFT, first visible output, and final completion; use a controlled clock.
• Early validation failures, configuration errors, MCP failure results, notification rejection, session errors, cancellations, and abruptly ended streams cannot be reported as business success.
• Repeated reads and retries do not inflate distinct target counts. Identical issue numbers in different repositories and issues versus PRs remain distinct. Multi-target joins do not multiply token totals.
• Model assertions such as "updated the wiki" or "triaged 100 issues" cannot create confirmed-action counts without execution evidence.
• A synthetic sensitive-data canary placed in forbidden telemetry fields is absent from exported spans, logs, events, exceptions, URLs, and any added workflow telemetry. Intentional answer publication remains governed by the existing Actions privacy controls. Existing GitHub/App/wiki authorization safeguards remain unchanged.
• Summaries remain available when their diagnostic trace is deliberately not sampled. Export outages and buffer pressure do not delay user output or repeat business operations; their loss/incompleteness is observable. Graceful shutdown flush is bounded for each enabled exporter/process.
• A nonproduction integration check demonstrates accepted native/event-derived spans, correct request correlation, unsampled BI records, trace drill-down, and the workbook on the actual pinned deployment versions.

Delivery plan and boundaries

Compatibility spike: Identify the deployed SDK, bundled CLI, AgentServer core, observability distro, telemetry resource, exporters, and current live tables. Confirm the intended environment/resource before querying it. Verify both protocols, per-turn parenting, usage fields, and the existing Python export path. Separately assess whether native CLI export can be enabled without incompatible tables, metric formats, duplicate exports, or new unsupported infrastructure.

MVP: Shared live event observer and host instrumentation; verified model/tool/subagent coverage through the existing Python pipeline; unsampled run/agent/target summaries; safe metadata and content-log gating; tests; Workbook and KQL definitions; setup and operating documentation. Use native CLI detail in the MVP only if the spike verifies a suitable export route.

Follow-on: Pricing-aware cost estimates, stronger dependency detail, optional client timing, release comparison alerts, and curated Power BI datasets. Human-rated triage accuracy, duplicate precision, planning acceptance, and time saved need separately defined feedback/evaluation data; execution telemetry cannot establish them.

Likely implementation surfaces: main.py; a small shared telemetry module; GitHub MCP operation boundaries; requirements and both deployment manifests; .env.example and README; telemetry tests; and optional Actions correlation that preserves existing privacy/publication controls.

Non-goals: New agent permissions, new model-facing tools, broader GitHub access, changed command semantics, raw transcript retention, an audit-grade exactly-once ledger, or automatic deployment.

Best-practice references

• Copilot SDK OpenTelemetry and v1.0.7 event contract: native instrumentation, per-request context propagation, and live event availability.
• Foundry hosted-agent telemetry configuration: project integration and runtime-managed exporters.
• OpenTelemetry GenAI metrics: common measurement semantics and evolving schema.
• Azure Monitor sampling and OpenTelemetry configuration: separate metrics/facts from sampled diagnostics and verify log behavior.
• Azure native OTLP ingestion: documented collector-mediated export and its separate endpoint/authentication requirements.
• Azure Workbooks data sources and Power BI integration: operational dashboards first, curated BI later.

Start with the client/session wiring, invocation and chat handlers in main.py, then review stream_output.py and github.py at the linked boundaries. Read the cited Copilot SDK and Foundry telemetry contracts before proposing the design. Done means an agreed, versioned architecture covering run accounting, diagnostic correlation, privacy, sampling, and verified BI records without authorizing deployment.