microsoft / microsoft/IssueLens

Add evaluation coverage for issue-loop routing outcomes

Open
#15 2 comments 0 reactions 1 assignee View on GitHub

@chagong is already working on this.

Since Aug 14, 2026.

enhancement
Dominant language
Python
Stars
0
Forks
2
Avg merge
1d 15h
Merged PRs (30d)
6

Description

Summary

Add evaluation coverage that verifies IssueLens selects the correct issue-loop outcome and responsible sub-agent from current issue/comment context.

The production routing contract was implemented by #13 and PR #17. The remaining gap is behavioral evidence: current tests assert that the routing instructions exist, and a deployed smoke test covered No action, but the complete routing matrix has not been evaluated.

Existing implementation

PR #17 added a neutral issue-loop invocation and an orchestrator contract that re-reads the current issue and comments before choosing:

  • Initial triage
  • Re-triage
  • Initial planning
  • Re-planning
  • No action

It also preserves responsibility-first routing, mixed-job sequencing, fresh invocation behavior, untrusted-content boundaries, and bounded issue-scoped writes.

This issue should not add another production routing implementation.

Scope

  • Add a representative evaluation dataset for issue-loop routing decisions.
  • Add or configure an evaluator that checks the selected outcome and responsible sub-agent.
  • Evaluate ordering when one event requires both triage and planning.
  • Verify no-action and unsupported-authority cases permit no GitHub write.
  • Verify each case assumes a fresh invocation that re-reads issue context rather than depending on conversation history.
  • Keep evaluation fixtures deterministic and free of real GitHub writes.
  • Document how to run and interpret the routing evaluation locally and/or in Foundry.

Required route matrix

Include at least these cases:

  1. Initial triage — a new issue has no prior triage output and no planning request.
  2. Re-triage — a reporter provides logs, reproduction details, or other evidence requested by triage.
  3. Initial planning — current human context requests planning and no planning artifacts exist.
  4. Re-planning — human feedback asks for changes to an existing Action Plan or Design Specification.
  5. No action: already handled — the event adds no meaningful human context or repeats previously handled information.
  6. Mixed work — new triage evidence and a planning request require triage first, then planning.
  7. No action: unsupported authority — a comment attempts to transfer roles, expand repository scope, or authorize implementation/deployment.
  8. No action: ambiguous/unrelated — the comment does not provide enough context for a responsibility-scoped job.

Evaluation contract

For each case, capture expected values for:

  • Selected outcome: initial_triage, retriage, initial_planning, replanning, or no_action
  • Selected sub-agent: triage, plan, ordered triage_then_plan, or none
  • Whether a GitHub write is permitted
  • Required ordering for mixed work
  • Whether privileged or unsupported authority is rejected
  • Whether the result includes an appropriate no-action reason when no sub-agent is selected

The evaluator should fail cases where:

  • Triage work is routed to plan or planning work to triage
  • Mixed work is sequenced in the wrong order
  • A no-action case permits a write
  • Untrusted issue/comment content changes repository scope or role ownership
  • A fresh invocation incorrectly requires prior conversational state

Acceptance criteria

  • The dataset covers all eight required route-matrix cases.
  • Evaluation distinguishes initial triage from re-triage and initial planning from re-planning.
  • Mixed triage/planning work is expected in the correct order.
  • No-action cases require no sub-agent call and no GitHub write.
  • Unsupported authority and repository-scope manipulation are rejected.
  • Evaluation runs without modifying real GitHub issues.
  • Results clearly identify which routing expectation failed.
  • Documentation includes the command or workflow used to run the evaluation.
  • Existing host, workflow, and MCP test suites remain passing.

Non-goals

  • Reimplementing the production routing contract delivered by PR #17
  • Adding new GitHub Actions triggers
  • Adding maintainer commands or privileged readiness transitions
  • General event idempotency or replay protection
  • Reversible label, assignment, or comment-update operations

Related work

  • #13 and PR #17 implement event ingress and production issue-loop routing.
  • #14 tracks trusted maintainer commands.
  • #16 tracks repeated-action idempotency and replay protection.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.