microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: azure-prepare – Terraform recipe produces unexpected Bicep/AZD files [Assertion mismatch]

Open
#1,267 2 comments 0 reactions 1 assignee Claimed by @wbreza View on GitHub
bug deploy skills test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Create a simple todo web app and deploy to Azure with Terraform as the infrastructure provider.
```

Follow-up: `Go with recommended options and proceed with Azure deployment.`

## Summary

**Run:** [Integration Tests - all #84](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23050118229)
**Test:** `azure-prepare_ - Integration Tests › prepare-deployment › creates correct files for Terraform recipe`
**Result:** Fail
**Duration:** 12m 5s

## Root Cause Category

**Assertion mismatch**

## Diagnosis

### What was expected
The test calls `expectFiles` asserting:
- **Must exist:** `plan.md`, `infra/*.tf` (Terraform files)
- **Must NOT exist:** `.bicep` files, `azure.yaml`

This verifies the agent uses the Terraform recipe and does NOT create Bicep/AZD files.

### What actually happened
`expectFiles` failed on the "must NOT exist" side — `expect(hasFile(files, pattern)).toBe(false)` received `true`. A `.bicep` or `azure.yaml` file was created alongside the Terraform files. The Terraform recipe was generated but also produced AZD or Bicep artifacts that should not be present.

### Why it failed
The `azure-prepare` skill created mixed output — both Terraform files (`infra/*.tf`) and either `azure.yaml` or `.bicep` files. This suggests the skill may be generating an AZD scaffold on top of the Terraform template, or a cleanup step is not removing unexpected files.

### Suggested fix
- Review the Terraform recipe template in `azure-prepare` to ensure it does not generate `azure.yaml` or `.bicep` files
- Add explicit cleanup/validation in the skill to remove non-Terraform IaC artifacts when Terraform is the chosen provider

## azure-prepare Skill Invocation

| Skill | Invoked |
|-------|---------|
| **azure-prepare** | **Yes** |

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

_Skill report artifact was not accessible in this analysis run. See run artifacts at https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23050118229/artifacts/5911858622_

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** v24.14.0
- **Model:** claude-sonnet-4.5 (or as overridden)
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23050118229
- **Commit:** `fef6de1232d9a1725537d8058e39e382e78233ab`

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23055952855) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.