microsoft / microsoft/IssueLens
Add evaluation coverage for issue-loop routing outcomes
@chagong is already working on this.
Since Aug 14, 2026.
- Dominant language
- Python
- Stars
- 0
- Forks
- 2
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 6
Description
Summary
Add evaluation coverage that verifies IssueLens selects the correct issue-loop outcome and responsible sub-agent from current issue/comment context.
The production routing contract was implemented by #13 and PR #17. The remaining gap is behavioral evidence: current tests assert that the routing instructions exist, and a deployed smoke test covered No action, but the complete routing matrix has not been evaluated.
Existing implementation
PR #17 added a neutral issue-loop invocation and an orchestrator contract that re-reads the current issue and comments before choosing:
- Initial triage
- Re-triage
- Initial planning
- Re-planning
- No action
It also preserves responsibility-first routing, mixed-job sequencing, fresh invocation behavior, untrusted-content boundaries, and bounded issue-scoped writes.
This issue should not add another production routing implementation.
Scope
- Add a representative evaluation dataset for issue-loop routing decisions.
- Add or configure an evaluator that checks the selected outcome and responsible sub-agent.
- Evaluate ordering when one event requires both triage and planning.
- Verify no-action and unsupported-authority cases permit no GitHub write.
- Verify each case assumes a fresh invocation that re-reads issue context rather than depending on conversation history.
- Keep evaluation fixtures deterministic and free of real GitHub writes.
- Document how to run and interpret the routing evaluation locally and/or in Foundry.
Required route matrix
Include at least these cases:
- Initial triage — a new issue has no prior triage output and no planning request.
- Re-triage — a reporter provides logs, reproduction details, or other evidence requested by triage.
- Initial planning — current human context requests planning and no planning artifacts exist.
- Re-planning — human feedback asks for changes to an existing Action Plan or Design Specification.
- No action: already handled — the event adds no meaningful human context or repeats previously handled information.
- Mixed work — new triage evidence and a planning request require triage first, then planning.
- No action: unsupported authority — a comment attempts to transfer roles, expand repository scope, or authorize implementation/deployment.
- No action: ambiguous/unrelated — the comment does not provide enough context for a responsibility-scoped job.
Evaluation contract
For each case, capture expected values for:
- Selected outcome:
initial_triage,retriage,initial_planning,replanning, orno_action - Selected sub-agent:
triage,plan, orderedtriage_then_plan, or none - Whether a GitHub write is permitted
- Required ordering for mixed work
- Whether privileged or unsupported authority is rejected
- Whether the result includes an appropriate no-action reason when no sub-agent is selected
The evaluator should fail cases where:
- Triage work is routed to
planor planning work totriage - Mixed work is sequenced in the wrong order
- A no-action case permits a write
- Untrusted issue/comment content changes repository scope or role ownership
- A fresh invocation incorrectly requires prior conversational state
Acceptance criteria
- The dataset covers all eight required route-matrix cases.
- Evaluation distinguishes initial triage from re-triage and initial planning from re-planning.
- Mixed triage/planning work is expected in the correct order.
- No-action cases require no sub-agent call and no GitHub write.
- Unsupported authority and repository-scope manipulation are rejected.
- Evaluation runs without modifying real GitHub issues.
- Results clearly identify which routing expectation failed.
- Documentation includes the command or workflow used to run the evaluation.
- Existing host, workflow, and MCP test suites remain passing.
Non-goals
- Reimplementing the production routing contract delivered by PR #17
- Adding new GitHub Actions triggers
- Adding maintainer commands or privileged readiness transitions
- General event idempotency or replay protection
- Reversible label, assignment, or comment-update operations
Related work
- #13 and PR #17 implement event ingress and production issue-loop routing.
- #14 tracks trusted maintainer commands.
- #16 tracks repeated-action idempotency and replay protection.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.