PostHog / PostHog/wizard

Make dynamic skill-menu failures deterministic across Anthropic and pi

Open
#1,033 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
197
Forks
51
Avg merge
1d 23h
Merged PRs (30d)
94

Description

Problem

Dynamic-skill programs depend on load_skill_menu, but a failed menu fetch produces different and model-dependent behavior across the two harnesses.

On the Anthropic path, createWizardToolsMcpServer prefetches skill-menu.json to build a Zod enum. When the fetch fails, the enum falls back to only integration. A valid call such as load_skill_menu { category: "feature-flags" } is then schema-rejected before the handler can return an actionable menu error.

On the pi path, the category is an unrestricted string, but a failed fetch returns the plain tool result Error: could not load the skill menu. Neither path is converted by the host into a terminal Wizard error. The agent must interpret the tool failure and emit the program's configured [ABORT] signal. That usually works when the prompt names this case, but it is not deterministic and is not covered by a failure-path test.

Why this matters

Programs without a fixed skillId, including ai-observability and feature-flags, cannot proceed without the menu. A transient registry outage should fail once with a useful explanation. It should not depend on the selected model to translate a schema error, retry, or eventually stop for another reason.

Relevant code

  • src/lib/wizard-tools/mcp.ts:157-169 - failed prefetch leaves categoryNames as ['integration'].
  • src/lib/wizard-tools/mcp.ts:340-369 - Anthropic schema and handler.
  • src/lib/agent/runner/harness/pi/tools.ts:86-112 - pi fetch and error result.
  • src/lib/agent/runner/sequence/linear.ts:142-263 - terminal error routing has no skill-menu failure case.

Suggested direction

Keep the category input callable even when startup discovery fails, then represent registry unavailability explicitly. The exact implementation can stay shared-tool-level, but Anthropic and pi should expose the same failure semantics and the runner should not rely on agent wording to finish the run.

Acceptance

  • A failed skill-menu.json fetch does not narrow Anthropic's category schema to an unrelated fallback value.
  • Anthropic and pi return equivalent structured failures when the menu is unavailable.
  • A dynamic-skill program terminates with one actionable error rather than relying on model retries or prose.
  • Focused tests cover menu-fetch failure for both harnesses.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failure paths in src/lib/wizard-tools/mcp.ts and src/lib/agent/runner/harness/pi/tools.ts, then trace terminal routing in src/lib/agent/runner/sequence/linear.ts. Run the focused harness tests if available and add coverage for failed skill-menu fetches in both Anthropic and pi. Done means both harnesses expose equivalent structured failures and a dynamic-skill run terminates with one actionable error.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.