microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-validate – containerized Container Apps deployment [Skill not invoked]

Open
#2,101 1 comment 0 reactions 1 assignee View on GitHub

@JasonYeMSFT is already working on this.

Since Apr 30, 2026.

azure-validate bug integration-test
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Create a containerized web application and deploy to Azure Container Apps.
```

## Summary

**Run:** [Integration Tests - all #198](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/25108826699)
**Test:** `azure-validate_ - Integration Tests › deployment-validation › terminates at validation for containerized web app on Container Apps`
**Result:** Fail
**Duration:** 1m 12s

## Root Cause Category

**Skill not invoked**

## Diagnosis

### What was expected

The test asserts two things:
1. `deploymentCommandRan` should be `false` — meaning `azd up` / `azd deploy` should **not** run before validation
2. `validateInvoked || validationCommandRan` should be `true` — azure-validate should be invoked (or a validation command run) before deployment

The test expects the agent to invoke azure-prepare (prepare code/infra) → invoke azure-validate (validate before deploying) → stop at that point.

### What actually happened

The agent correctly invoked `azure-prepare` first and generated a complete TypeScript/Node.js application (Express.js framework), complete with a multi-stage Dockerfile, Bicep infrastructure templates, Application Insights monitoring, health probes, and auto-scaling configuration. The agent then immediately ran `azd up` to begin deployment — bypassing the azure-validate skill entirely.

Failure at `tests/azure-validate/integration.test.ts:214`:
```
Expected: false
Received: true
// deploymentCommandRan = true because agent ran `azd up`
````

### Why it failed

The `azure-prepare` skill's workflow ran `azd up` directly from within its flow, without pausing to invoke `azure-validate`. The azure-prepare skill's deployment plan is expected to include a validation phase (invoking azure-validate) before executing deployment. This failure matches the same root cause as the related static whiteboard issue (#2054): azure-prepare proceeds straight to `azd up` instead of routing to azure-validate first.

### Suggested fix

- Update the azure-prepare skill workflow to explicitly invoke `azure-validate` **before** running any `azd up` / `azd deploy` commands when deployment is requested.
- Alternatively, add a guard in azure-prepare that prevents running deployment commands without completing a validation step first.

## azure-validate Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-validate** | No |

The agent invoked azure-prepare correctly, but azure-prepare ran `azd up` directly without chaining to azure-validate.

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

### Test 3: deployment-validation_terminates_at_validation_for_containerized_web_app_on_Container_Apps

**Type:** End-to-End
**Prompt:** "Create a containerized web application and deploy to Azure Container Apps."
**Runs:** 1 | **Pass Rate:** 100% (per skill report) | **Avg Confidence:** 95%

**What Happened:**
The agent invoked azure-prepare and generated a complete TypeScript/Node.js application with Express.js framework. All deployment artifacts were created including multi-stage Dockerfile, Bicep infrastructure templates, and azure.yaml configuration. The agent initiated `azd up` to begin deployment.

**✅ What Went Well:**
- azure-prepare skill correctly identified and invoked immediately
- Complete application stack generated with production-ready patterns (Node.js 20, TypeScript, multi-stage Docker build)
- Comprehensive Bicep infrastructure with Application Insights monitoring
- Managed identity security model applied throughout
- Auto-scaling configured (1-3 replicas)

**❌ What Went Wrong:**
- Deployment command (`azd up`) ran before azure-validate was invoked
- `deploymentCommandRan` assertion failed (expected `false`, received `true`)

> Note: The SKILL-REPORT assessed this as 100% pass (soft-pass based on task completion), but the JUnit hard assertion `expect(deploymentCommandRan).toBe(false)` failed because `azd up` ran.

## Environment

- **Runner OS:** ubuntu-latest
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/25108826699
- **Commit:** `13345dedd7cc6cde2b1f9fc25e829bfa4b648071` (Add entra-agent-id skill)

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/25119882862/agentic_workflow) · ● 5.7M · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.