Automattic / Automattic/skillsmith

Skip misconfigured agents across all phases of a run

Open
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Goal

When `skillsmith` runs in a project whose `skillsmith.config.ts` defines agents, any agent that is misconfigured — e.g. an invalid API token, a provider that doesn't exist, or similar configuration errors — is skipped for the entire run rather than causing failures or being partially executed. A skipped agent does no work in any phase: testers don't generate code, judges don't review, and the agent is excluded from evaluation in subsequent iterations. In effect, misconfigured agents are ignored everywhere for that run.

## Constraints

- Hooks must be informed about misconfigured agents — either the set of agent definitions that failed, or the filtered list of agents that will actually run (whichever shape makes more sense; the exact contract is open for the design phase).

## Assumptions / directions to explore

- The CLI already announces failing agents today. *(Open — to be confirmed; the desired behavior is that failures continue to be surfaced in the CLI.)*

Contributor guide

Open the contributing guide

Research direction

Start by tracing how skillsmith reads skillsmith.config.ts, validates agents, announces failures in the CLI, and passes agents through each run phase and hook. Define how failed agents are represented or filtered, then verify that they perform no work in any phase, are excluded from later evaluation, and remain surfaced to hooks and the CLI.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.