microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-rbac – missing Bicep role assignment code for Key Vault secret access [Assertion mismatch]

Open
#1,542 0 comments 0 reactions 1 assignee Claimed by @msalaman View on GitHub
bug integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

> "What role do I need to read secrets from Azure Key Vault?"

## Summary

**Run:** [Integration Tests - all #122](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23645613049)
**Test:** `azure-rbac_ - Integration Tests › azure-rbac › recommends Key Vault Secrets User for secret access`
**Result:** Fail
**Duration:** 50s

## Root Cause Category

**Assertion mismatch**

## Diagnosis

### What was expected
The test checks four conditions for a complete Key Vault secret role recommendation response:
1. `azure-rbac` skill was invoked ✅
2. Response mentions "Key Vault Secrets User" ✅ (likely)
3. Response includes `az role assignment` CLI command ✅ (likely)
4. Response includes `Microsoft.Authorization/roleAssignments` Bicep code ❌

```
Error: expect(received).toBe(expected) // Object.is equality
Expected: true
Received: false
at tests/azure-rbac/integration.test.ts:137:30
```

The assertion at line 137 is `expect(hasBicepCode).toBe(true)`, where `hasBicepCode` checks `doesAssistantMessageIncludeKeyword(agentMetadata, "Microsoft.Authorization/roleAssignments")`.

### What actually happened
The agent invoked `azure-rbac`, mentioned the Key Vault Secrets User role, and likely provided CLI commands — but did **not** include a Bicep code snippet for `Microsoft.Authorization/roleAssignments` in its response. The response was incomplete from the test's perspective.

### Why it failed
For a "what role do I need" discovery query, the agent may have optimized for a concise textual answer (role name + CLI command) without generating the full Bicep infrastructure-as-code template. The skill's output completeness guidance may not require Bicep for informational queries.

### Suggested fix
- Update the `azure-rbac` skill to always include a Bicep role assignment snippet for any role recommendation response
- Ensure the skill's instructions explicitly state: "For every RBAC role recommendation, provide: (1) role name, (2) `az role assignment create` CLI command, AND (3) Bicep `Microsoft.Authorization/roleAssignments` resource"
- This pattern is already working for other azure-rbac tests (e.g., `provides Bicep code for role assignment` passes); apply same logic to discovery queries

## azure-rbac Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-rbac** | **Yes** (skill invoked, partial response) |

## Skill Report Context

**From SKILL-REPORT.md:**
- Skill Invocation Success Rate: **100%** (10/10)
- Overall Test Pass Rate: **100%** (14/14) ← Note: this reflects pre-run assessment; actual junit shows 1 failure
- Average Confidence: **97%**

The SKILL-REPORT shows perfect 100% for azure-rbac, but the test at line 137 checking for Bicep code in the Key Vault secret access response failed the assertion.

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.x
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23645613049
- **Commit:** `e9285b05c936a03a0450f1939fef7fb69472e6e5`
- **Trigger:** schedule

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23650220396) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.