Azure / Azure/azure-sdk-for-python
[azure-ai-projects] Evaluation sample sample_agent_evaluation.py has bug
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: azure-ai-projects
- **Package Version**:
- **Operating System**:
- **Python Version**:
**Describe the bug**
When run this sample, error: **INVALID VALUE: (UserError) Missing inputs for line 1: 'data.response'**. This is because the data mapping is wrong: `"response": "{{item.response}}"}`, thus actually there is no `response` in `item`. The correct mapping should be `"response": "{{sample.output_text}}"`.
BTW, I have one question for custom code-based evaluator. Do you have a full documentation on the custom evaluators? I wonder how the parameters should be and how to get responses of agent. I ran an evaluation with custom code based evaluator targeted on an agent, but I cannot get the responses of the agent. I tried `sample.get("output_text")`, `sample.output_text`. Code of my evaluator:
```python
def grade(sample, item) -> float:
try:
response = sample.get("output_text")
return 1.0 if len(response) < 100 else 0.0
except:
return 0.1
```
**To Reproduce**
Steps to reproduce the behavior:
1.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.