feat: add routing-focused eval support
- Dominant language
- Go
- Stars
- 894
- Forks
- 68
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
## Motivation
`skill-up` is good at evaluating whether an Agent completes a task correctly once a Skill is installed. For Skill development, another important question is whether the Skill should be triggered for a prompt at all.
Routing behavior matters because:
- a Skill should trigger for intended prompts
- it should not trigger for unrelated prompts
- in multi-Skill environments, one Skill should not over-trigger or shadow another Skill
## Proposed capability
A routing case could look like this:
```yaml
id: route-code-review-positive
tag: routing
input:
prompt: "Review this pull request and identify risks"
routing_expect:
must_include: [code-review]
allow_extra: false
max_skills: 1
```
A negative case could look like this:
```yaml
id: route-code-review-negative
tag: routing
input:
prompt: "Write a poem about winter"
routing_expect:
must_not_include: [code-review]
```
Possible semantics:
- `must_include`: expected Skill names must be selected or loaded
- `must_not_include`: listed Skills must not be selected or loaded
- `allow_extra: false`: no other Skills should be triggered
- `max_skills`: upper bound for triggered Skills
## Why this matters
Without routing evals, a Skill can pass positive task cases but still be harmful in practice:
- it may trigger too often
- it may respond outside its intended scope
- it may cause negative delta in multi-Skill setups
## Initial small scope
To keep the first version manageable, this could start as:
1. model routing cases as normal cases with `tag: routing`
2. add a `routing_expect` schema
3. support only engines that expose triggered/loaded Skill information
4. return a clear skipped/unsupported result when routing info is unavailable
## Open questions
1. Do current agent adapters expose enough information to reliably identify which Skills were triggered?
2. Should routing be a top-level suite type, or regular cases with `tag: routing` plus `routing_expect`?
3. Should the first version support only single-Skill routing, leaving multi-Skill routing for later?
Contributor guide
Research direction
Start by reviewing the current case schema and agent adapter interfaces to see how cases are modeled and whether triggered or loaded Skills are exposed. Compare the proposed routing_expect fields and determine what a skipped or unsupported result would require; done means the scope and adapter support are defined clearly enough to implement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100