microsoft / microsoft/GitHub-Copilot-for-Azure

Integration test failure: microsoft-foundry – foundry agent invoke [Skill not invoked]

Open
#1,404 2 comments 0 reactions 3 assignees Claimed by @tendau View on GitHub
bug foundry integration-test test-failure
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Prompt

```
Send a test message to my Foundry agent
```

## Summary

**Run:** [Integration Tests - all #105](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23330233290)
**Test:** `microsoft-foundry_ - Integration Tests › invokes skill for relevant prompt`
**Result:** Fail
**Duration:** 0m 44.6s

## Root Cause Category

**Skill not invoked**

## Diagnosis

### What was expected

The test asserts that the `microsoft-foundry` skill is invoked when given a prompt asking to send a test message to a Foundry agent. Specifically, `isSkillInvoked(agentMetadata, "microsoft-foundry")` should return `true`.

### What actually happened

The agent responded to the prompt "Send a test message to my Foundry agent" but did not invoke the `microsoft-foundry` skill. The test times out after ~44 seconds of agent activity (179,048 input tokens / 1,332 output tokens across 4 API calls) with the skill invocation check returning `false`.

### Why it failed

The prompt "Send a test message to my Foundry agent" is phrased ambiguously — it could be interpreted as a general question about how to send a message rather than a direct invocation of the foundry agent invoke sub-skill. The agent likely provided guidance on using the tool without triggering the actual skill invocation. The same test (`invokes skill for relevant prompt`) passes in two other test suites (`foundry-agent/deploy` at 10.7s and `foundry-agent/integration.test.ts` at 12.5s), suggesting the prompt context or test configuration in the `invoke` sub-directory differs.

### Suggested fix

- Review the `foundry-agent/invoke` skill triggers/description to ensure "send a test message to my Foundry agent" reliably matches the skill's activation criteria
- Consider making the prompt more explicit, e.g. "Use the microsoft-foundry skill to invoke my Foundry agent with a test message"
- Investigate why the identical prompt passes in other test suites but fails specifically in `foundry-agent/invoke/integration.test.ts`

## microsoft-foundry Skill Invocation

| Skill | Invoked |
|-------|---------|
| **microsoft-foundry** | No |

## Skill Report Context

Per-test section from SKILL-REPORT.md (click to expand)

**Note:** The full SKILL-REPORT.md artifact could not be retrieved directly; the following is inferred from job logs.

- **Test suite:** `microsoft-foundry_ - Integration Tests` (foundry-agent/invoke/integration.test.ts)
- **Test suite result:** 1 failed, 1 passed (2 total)
- **Failing test:** `invokes skill for relevant prompt` (44.6s)
- **Token usage for test run:** 37,766 in / 325 out | 1 API call | Duration: 6.0s (final assertion run)
- **Overall run result:** 66/68 tests passed (97.1%)
- **Overall test suite processed:** 55 test report directories, 14 test suites

## Environment

- **Runner OS:** ubuntu-latest
- **Node.js:** 22.x (from workflow)
- **Model:** claude-sonnet-4.5
- **Run URL:** https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23330233290
- **Commit:** 878863eb14a0b2b316492411b2ad67d777aec04c
- **Test file:** `tests/microsoft-foundry/foundry-agent/invoke/integration.test.ts:37`

> Generated by [Analyze Test Run](https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/23331482299) · [◷](https://github.com/search?q=repo%3Amicrosoft%2FGitHub-Copilot-for-Azure+is%3Aissue+%22gh-aw-workflow-call-id%3A+microsoft%2FGitHub-Copilot-for-Azure%2Fanalyze-test-run%22&type=issues)

> [!WARNING]
>
> ⚠️ Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `productionresultssa7.blob.core.windows.net`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "productionresultssa7.blob.core.windows.net"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.