alibaba / alibaba/skill-up

feat: add routing-focused eval support

Open
#166 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
894
Forks
68
Avg merge
1d 15h
Merged PRs (30d)
38

Description

## Motivation

`skill-up` is good at evaluating whether an Agent completes a task correctly once a Skill is installed. For Skill development, another important question is whether the Skill should be triggered for a prompt at all.

Routing behavior matters because:

- a Skill should trigger for intended prompts
- it should not trigger for unrelated prompts
- in multi-Skill environments, one Skill should not over-trigger or shadow another Skill

## Proposed capability

A routing case could look like this:

```yaml
id: route-code-review-positive
tag: routing

input:
prompt: "Review this pull request and identify risks"

routing_expect:
must_include: [code-review]
allow_extra: false
max_skills: 1
```

A negative case could look like this:

```yaml
id: route-code-review-negative
tag: routing

input:
prompt: "Write a poem about winter"

routing_expect:
must_not_include: [code-review]
```

Possible semantics:

- `must_include`: expected Skill names must be selected or loaded
- `must_not_include`: listed Skills must not be selected or loaded
- `allow_extra: false`: no other Skills should be triggered
- `max_skills`: upper bound for triggered Skills

## Why this matters

Without routing evals, a Skill can pass positive task cases but still be harmful in practice:

- it may trigger too often
- it may respond outside its intended scope
- it may cause negative delta in multi-Skill setups

## Initial small scope

To keep the first version manageable, this could start as:

1. model routing cases as normal cases with `tag: routing`
2. add a `routing_expect` schema
3. support only engines that expose triggered/loaded Skill information
4. return a clear skipped/unsupported result when routing info is unavailable

## Open questions

1. Do current agent adapters expose enough information to reliably identify which Skills were triggered?
2. Should routing be a top-level suite type, or regular cases with `tag: routing` plus `routing_expect`?
3. Should the first version support only single-Skill routing, leaving multi-Skill routing for later?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the current case schema and agent adapter interfaces to see how cases are modeled and whether triggered or loaded Skills are exposed. Compare the proposed routing_expect fields and determine what a skipped or unsupported result would require; done means the scope and adapter support are defined clearly enough to implement.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.