microsoft / microsoft/GitHub-Copilot-for-Azure
Integration test failure: azure-rbac – missing Bicep role assignment code for Key Vault secret access [Assertion mismatch]
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Prompt
> "What role do I need to read secrets from Azure Key Vault?"
## Summary
**Run:** [Integration Tests - all #122](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23645613049)
**Test:** `azure-rbac_ - Integration Tests › azure-rbac › recommends Key Vault Secrets User for secret access`
**Result:** Fail
**Duration:** 50s
## Root Cause Category
**Assertion mismatch**
## Diagnosis
### What was expected
The test checks four conditions for a complete Key Vault secret role recommendation response:
1. `azure-rbac` skill was invoked ✅
2. Response mentions "Key Vault Secrets User" ✅ (likely)
3. Response includes `az role assignment` CLI command ✅ (likely)
4. Response includes `Microsoft.Authorization/roleAssignments` Bicep code ❌
```
Error: expect(received).toBe(expected) // Object.is equality
Expected: true
Received: false
at tests/azure-rbac/integration.test.ts:137:30
```
The assertion at line 137 is `expect(hasBicepCode).toBe(true)`, where `hasBicepCode` checks `doesAssistantMessageIncludeKeyword(agentMetadata, "Microsoft.Authorization/roleAssignments")`.
### What actually happened
The agent invoked `azure-rbac`, mentioned the Key Vault Secrets User role, and likely provided CLI commands — but did **not** include a Bicep code snippet for `Microsoft.Authorization/roleAssignments` in its response. The response was incomplete from the test's perspective.
### Why it failed
For a "what role do I need" discovery query, the agent may have optimized for a concise textual answer (role name + CLI command) without generating the full Bicep infrastructure-as-code template. The skill's output completeness guidance may not require Bicep for informational queries.
### Suggested fix
- Update the `azure-rbac` skill to always include a Bicep role assignment snippet for any role recommendation response
- Ensure the skill's instructions explicitly state: "For every RBAC role recommendation, provide: (1) role name, (2) `az role assignment create` CLI command, AND (3) Bicep `Microsoft.Authorization/roleAssignments` resource"
- This pattern is already working for other azure-rbac tests (e.g., `provides Bicep code for role assignment` passes); apply same logic to discovery queries
## azure-rbac Skill Invocation
| Skill | Invoked |
|-------|---------|
| **azure-rbac** | **Yes** (skill invoked, partial response) |
## Skill Report Context
**From SKILL-REPORT.md:**
- Skill Invocation Success Rate: **100%** (10/10)
- Overall Test Pass Rate: **100%** (14/14) ← Note: this reflects pre-run assessment; actual junit shows 1 failure
- Average Confidence: **97%**
The SKILL-REPORT shows perfect 100% for azure-rbac, but the test at line 137 checking for Bicep code in the Key Vault secret access response failed the assertion.
## Environment
- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23645613049
- **Commit:** `e9285b05c936a03a0450f1939fef7fb69472e6e5`
- **Trigger:** schedule
> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23650220396) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)
Contributor guide
Assessment
This issue has not been assessed yet.