microsoft / microsoft/GitHub-Copilot-for-Azure
Add skill-routing integration tests to PR CI pipeline
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Background
PR #1210 introduced a skill-routing regression (#1599) that went undetected for **20 days** because integration tests only run on a nightly schedule, not on PRs.
## Problem
- `test-all-integration.yml` runs only on `schedule` (nightly cron) and `workflow_dispatch` (manual)
- `test-all-skills.yml` runs on PRs but only runs unit/trigger tests, which don't test LLM-based routing
- The failing scenario (generic deploy prompt + codebase context → wrong skill wins) can only be caught by integration tests
## Proposal
Add a **lightweight subset** of integration tests to the PR CI pipeline:
1. **Skill-routing smoke tests** — For each skill that has `PREFER OVER` or `DO NOT USE FOR` clauses, run the integration test scenarios that validate correct routing
2. **Triggered only when skill files change** — Use `paths:` filters to only run when `plugin/skills/**` or `tests/**/integration.test.ts` are modified
3. **Parallel matrix** — Run only affected skills, not the full matrix
### Example workflow trigger
```yaml
on:
pull_request:
paths:
- 'plugin/skills/**'
- 'tests/**/integration.test.ts'
```
## Impact
This would have caught #1599 at PR #1210 review time, preventing the 20-day detection delay.
## Related
- Root cause: #1599
- Fix: #1644
Contributor guide
Research direction
Start by comparing the triggers and test scope in test-all-integration.yml and test-all-skills.yml. Review the integration tests under tests/**/integration.test.ts and the skill files under plugin/skills/**, then determine how to select affected skills in a parallel PR matrix. Done means routing smoke tests run for relevant changes, while unrelated pull requests retain the existing CI behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100