PostHog / PostHog/wizard

07 — Rollout + per-runner observability (canary via wizard-runner)

Open
#527 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
197
Forks
51
Avg merge
1d 23h
Merged PRs (30d)
94

Description

Epic: #520 · Behavior change: first user-visible · Depends on: #521–#526

Summary

Add per-pair guardrail telemetry and start a small canary using the existing wizard-runner flag.

Scope

  • Tag run telemetry with the resolved router + runner + model: success rate, wall-clock, tool-error rate,
    token usage, YARA blocks. (Cuts by runner and by model both fall out.)
  • Structured run logging to file (the TUI owns the console): assistant turns + tool I/O.
  • A dashboard comparing runners (anthropic vs pi) and routers.
  • Start a small canary by ramping the existing wizard-runner toward pi for a targeted slice, gated on
    metrics. The one-flip revert is wizard-runner=anthropic (today's default) — no new kill-switch flag.

Acceptance criteria

  • Telemetry carries the resolved router + runner + model; dashboard live.
  • Canary active via wizard-runner; revert to anthropic verified.
  • Guardrails at/above the anthropic baseline before any ramp.

Files

  • src/lib/agent/runner/*, analytics

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing runner code under src/lib/agent/runner/*, the analytics area, and the current wizard-runner flag behavior. Define the telemetry and structured logging points, then verify the dashboard compares runners and routers, the canary ramps toward pi, and reverting wizard-runner=anthropic works with guardrails at the anthropic baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
analytics, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.