microsoft / microsoft/hve-core

refactor(hooks): retire the telemetry hook in favor of opt-in copilot-otel-metrics capture

Open
#2,717 2 comments 0 reactions 1 assignee View on GitHub

@WilliamBerryiii is already working on this.

Since Aug 14, 2026.

refactor security skills
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Summary

Retire the local telemetry hook and its owned runtime, and let the `copilot-otel-metrics` skill be the single supported path for Copilot telemetry capture.

The hook has been repaired four times for cross-platform defects (#2532, #2538, #2540, and #2352), and #2496 is still open against it. Rather than repair it a fifth time or reduce its surface as proposed in #2562, this retires the collection mechanism whose design is the source of both the reliability and the privacy exposure.

## The privacy case

This is the primary motivation, not a side effect.

**Collection was ambient, not consented.** The manifest registered one command against eleven lifecycle events: `sessionStart`, `userPromptSubmitted`, `userPromptSubmit`, `preToolUse`, `postToolUse`, `subagentStart`, `subagentStop`, `sessionEnd`, `stop`, `agentStop`, and `preCompact`. Installing the plugin enrolled the developer in collection across every prompt, every tool call, and every subagent dispatch. Consent was a property of installation, not a decision the developer made about their own session data.

The skill inverts this. Nothing is captured until the developer enables an editor setting, and the skill presents the exact diff before writing it. Someone who installs the package and never opts in emits nothing.

**Ambient collection wrote a raw input artifact.** The hook runtime produced `raw-input.jsonl` on the local disk. That file recorded session input ahead of any filtering decision, so the privacy boundary sat after collection rather than before it.

The replacement puts the boundary first. `examples/otel-collector-local.yaml` applies a fail-closed `redaction` processor across the trace, metric, and log pipelines, with `allow_all_keys: false` making `allowed_keys` authoritative and `blocked_values` running as a second independent pass over whatever survives. An attribute introduced by a future extension release is dropped by default rather than stored by default.

**Filtering is measured, not asserted.** `tests/test_collector_carriers.py` starts the pinned Collector with the shipped configuration, places a distinct marker in every OTLP carrier the signal model can transport, and records what actually survives. A carrier that opens fails the test rather than passing quietly. The privacy claim is backed by evidence rather than by design intent.

**The residual exposure is documented rather than papered over.** `SECURITY.md` is a full STRIDE model across seven trust boundaries with sixteen registered gaps. `G-INF-1` states plainly that spans carry prompt text, tool call arguments and results, and system instructions on a configuration left at its documented default, and that the skill cannot change extension behavior. The local path drops unknown content attributes at the Collector and the Azure path deletes the seven observed content attributes before export, but the plaintext loopback hop is named as still outside the skill's control. Honest disclosure of what the control does not reach is worth more than a claim of complete coverage.

**Telemetry was in the critical path.** Hooks are fail-closed, so a telemetry defect could deny every tool call in a session, which is exactly what #2496 reports. Ancillary observability code held the power to halt all work. Removing it from the hook path removes that class of failure entirely, and it also removes the incentive to weaken the fail-closed posture to accommodate telemetry.

## Relationship to open issues

* **Resolves #2496.** The Windows PowerShell 5.1 fail-closed denial is a hook defect. Deleting the hook removes the failure mode rather than patching it.
* **Supersedes #2562.** That issue proposes reducing telemetry to a three-hook spine while keeping `sessionStart` as the opt-in consent gate. This goes further and removes the hook entirely, because the editor setting is a better consent gate than a hook: it is owned by the developer, visible in their own settings, and revocable without touching the package.

The trade-off #2562 raises still applies and is accepted here. Some report surfaces become dependent on what the developer chose to enable, which is the intended consequence of moving from ambient to opt-in collection.

## Capability changes

Two capabilities are removed rather than replaced, and the change should be reviewed with that in mind:

* Automatic hook-driven collection. Capture is now opt-in through an editor setting.
* Local JSONL to static HTML reporting. Reporting moves to Grafana dashboards and query helpers.

Fleet and organization capture through the Azure path is new capability with no hook equivalent.

## Scope

* Remove `.github/hooks/shared/telemetry.json` and its owned runtime and test package.
* Remove the dedicated local telemetry guide, with `docs/customization/copilot-otel-metrics.md` as the replacement.
* Drop the `hooks` key and its maturity entry from the affected package recipes.
* Register `copilot-otel-metrics` in the experimental package.
* Generalize plugin and extension test fixtures that referenced the real manifest to synthetic fixtures.

## Acceptance criteria

* [ ] No references to the deleted hook paths remain in active code, configuration, or documentation.
* [ ] `npm run lint:hooks`, `npm run plugin:validate`, `npm run validate:skills`, and `npm run docs:generate:check` pass.
* [ ] The skill ships `pyproject.toml`, `uv.lock`, and its test package so dependency scanning can resolve it.
* [ ] Removed capabilities are stated explicitly in the pull request body.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.