microsoft / microsoft/GitHub-Copilot-for-Azure

Remove deprecated Waza, Jest, and TriggerMatcher infrastructure

Open
#1,821 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

infrastructure integration-test untriaged
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

Background & Motivation
Once all migrations are validated and the unified CI pipeline is running, the old Waza infrastructure, Jest integration tests, and TriggerMatcher utility must be fully removed. Leaving deprecated code in the repo increases confusion, maintenance burden, and onboarding friction for new contributors.

User Story
As a Platform Engineer, I want all deprecated evaluation infrastructure removed so that new contributors see one evaluation framework and one CI pipeline, eliminating confusion about which system to use.

Solution Approach

  • Pre-flight validation: Before deleting anything, run the full Vally eval suite and confirm all migrated evals pass. Run the migration validation script from Issue 4 to confirm zero discrepancies.
  • Waza removal (FR-E1, FR-E4):
    • Delete all tasks/*.yaml directories under old eval locations
    • Delete .waza.yaml project config
    • Remove Waza-related npm scripts from package.json
    • Remove any Waza bridge scripts
  • Jest integration test removal (FR-E2):
    • Delete all integration.test.ts files from tests/<skill>/
    • Update Jest config to exclude removed test paths (FR-E6)
    • Keep tests/<skill>/unit.test.ts and tests/utils/ (minus deprecated utilities)
  • TriggerMatcher retirement (FR-E3):
    • Delete all triggers.test.ts files from tests/<skill>/
    • Delete tests/utils/TriggerMatcher.ts
    • Delete associated Jest snapshots (__snapshots__/)
  • CI cleanup (FR-E5):
    • Remove old eval.yml / Waza CI workflow
    • Remove old Jest integration CI workflow (or the integration-specific jobs)
    • Ensure the new evals.yml from Issue 4 is the sole eval CI pipeline
  • Documentation (FR-E7, FR-E8):
    • Update tests/README.md to reflect that LLM evals live in evals/, unit tests in tests/
    • Archive EVAL-MIGRATION-REPORT.md (move to docs/archive/)
  • Validation: After deletion, run vally eval --suite full and all remaining Jest unit tests to confirm nothing is broken.

Files to delete:

.waza.yaml
evals/*/tasks/                             (old Waza task directories)
tests/*/eval/                              (old Waza eval directories under tests/)
tests/*/integration.test.ts                (×38 files)
tests/*/triggers.test.ts                   (×34 files)
tests/*/evals/                             (old nested eval directories)
tests/*/__snapshots__/triggers.*           (trigger test snapshots)
tests/utils/TriggerMatcher.ts
.github/workflows/eval.yml                 (old Waza CI workflow, if separate)

Files to update:

package.json                               (remove Waza npm scripts)
jest.config.ts                             (exclude removed test paths)
tests/README.md                            (update to reflect new structure)
.github/workflows/                         (remove integration test jobs from old workflows)

Acceptance Criteria

  • FR-E1: All Waza task files deleted after migration validation confirms equivalence
  • FR-E2: All Jest integration test files deleted after migration validation
  • FR-E3: TriggerMatcher utility and all Jest trigger test files deleted
  • FR-E4: .waza.yaml, bridge scripts, and Waza npm scripts removed from package.json
  • FR-E5: Old CI workflows replaced by the unified evals.yml pipeline
  • FR-E6: Jest config updated to exclude removed test paths — remaining unit tests still pass
  • FR-E7: tests/README.md updated to document that LLM evals live in evals/ and unit tests stay in tests/
  • FR-E8: EVAL-MIGRATION-REPORT.md archived to docs/archive/
  • Zero grep hits for "waza" in the repo (excluding docs/archive) — per PRD success metric
  • Zero grep hits for "TriggerMatcher" in the repo (excluding docs/archive) — per PRD success metric
  • vally eval --suite full passes after cleanup
  • All remaining Jest unit tests pass after cleanup
  • No orphaned fixture directories or config files remain

Out of Scope

  • Writing new test cases to expand coverage (PRD §8)
  • Migrating Jest unit tests to any other framework (PRD §8)
  • Changes to SKILL.md files (PRD §8)

Testing Expectations

  1. Run vally eval --suite full — all evals pass.
  2. Run remaining Jest unit tests — all pass.
  3. Run grep -r "waza" . --include="*.ts" --include="*.yaml" --include="*.yml" --include="*.json" — zero hits outside docs/archive/.
  4. Run grep -r "TriggerMatcher" . --include="*.ts" — zero hits.
  5. Verify CI pipeline triggers correctly on a test PR after old workflows are removed.

Constraints

  • All deletions must happen AFTER migration validation passes (PRD success metric: 0 cases lost).
  • Jest config changes must not break remaining unit tests.

Related Issues

  • Depends on Issue 1 — Waza eval configs must be migrated before Waza files can be deleted.
  • Depends on Issue 2 — Jest integration tests must be migrated before deletion.
  • Depends on Issue 3 — Trigger tests and TriggerMatcher must be migrated before retirement.
  • Depends on Issue 4 — Unified CI pipeline must be validated before old CI workflows are removed.
  • This is the final phase — no other issues depend on it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.