Azure / Azure/azure-sdk-for-python

Agent Monitoring Continuous Evaluation for Custom Agents Failing with error "No trace data found"

Open
#47,812 4 comments 1 reaction 1 assignee Assigned to @posaninagendra View on GitHub
AI Projects Client customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: Agent Monitoring Continuous Evaluation
- **Package Version**:
- **Operating System**:
- **Python Version**:

**Describe the bug**
We have setup continuous monitoring via the UI for our custom agent, however evaluation run fails with this error even though traces exist:

**There was an error with this evaluation run.No trace data found where customDimensions.gen_ai.input.messages or customDimensions.gen_ai.output.messages contain data. Please check the agent running status for agent 'ifd-mono' during time range: from 2026-07-02T05:50:08+00:00 to 2026-07-02T06:50:08+00:00**

Image

Image

Image

**To Reproduce**
Set up monitoring for your custom agents
1. Onboard your custom agent to Foundry using the instructions in [Register and manage custom agents](https://learn.microsoft.com/en-us/azure/foundry/control-plane/register-custom-agent).
2. Instrument your agent to comply with the [semantic conventions for generative AI solutions in the OpenTelemetry standard](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
3. Configure your agent to send telemetry to the same Application Insights instance as your Foundry project to enable continuous evaluation features.
4. In Foundry Control Plane, go to the Asset page and select your agent.
5. Select the Monitor tab to view your metrics and charts.
6. Set up continuous evaluations using the methods outlined in [Configure settings](https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/how-to-monitor-agents-dashboard?tabs=python#configure-settings).

**Note: We only implemented the configure settings via the UI and didn't create the python implementation as we understood it achieves the same thing?**

**Expected behavior**
A clear and concise description of what you expected to happen.
Evalution result should appear in the Evaluation page successfully

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
See user_log attached

[user_logs (12).txt](https://github.com/user-attachments/files/29587816/user_logs.12.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.