Azure / Azure/azure-sdk-for-python

[azure-ai-projects] Evaluation sample sample_agent_evaluation.py has bug

Open
#44,306 4 comments 0 reactions 1 assignee Claimed by @w-javed View on GitHub
AI Projects customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: azure-ai-projects
- **Package Version**:
- **Operating System**:
- **Python Version**:

**Describe the bug**
When run this sample, error: **INVALID VALUE: (UserError) Missing inputs for line 1: 'data.response'**. This is because the data mapping is wrong: `"response": "{{item.response}}"}`, thus actually there is no `response` in `item`. The correct mapping should be `"response": "{{sample.output_text}}"`.

BTW, I have one question for custom code-based evaluator. Do you have a full documentation on the custom evaluators? I wonder how the parameters should be and how to get responses of agent. I ran an evaluation with custom code based evaluator targeted on an agent, but I cannot get the responses of the agent. I tried `sample.get("output_text")`, `sample.output_text`. Code of my evaluator:
```python
def grade(sample, item) -> float:
try:
response = sample.get("output_text")
return 1.0 if len(response) < 100 else 0.0
except:
return 0.1
```

**To Reproduce**
Steps to reproduce the behavior:
1.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.