awslabs / awslabs/Agent-EvalKit
metrics.py: unconditional exit() in create_test_case_from_trace prevents test case creation
- Dominant language
- Python
- Stars
- 38
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
In `examples/qa_agent_evaluation/eval/metrics.py`, `create_test_case_from_trace()` calls `exit()` unconditionally. It sits directly after the three debug `print()` calls and **before** both the `if not query or not answer:` guard and the `LLMTestCase(...)` construction:
```python
print(f" Extracted query: {query}")
print(f" Extracted answer: {answer}")
print(f" Extracted search results: {search_results}")
exit()
if not query or not answer:
...
test_case = LLMTestCase(input=query, actual_output=answer, retrieval_context=search_results)
return test_case
```
The function can therefore never return a test case — it terminates the interpreter on the first trace it processes, so `run_evaluation.py` cannot complete against the bundled example.
### Since when
`metrics.py` has a single commit in its path history: 151fb44 (`add qa agent evaluation example`, 2025-11-24). The `exit()` has been present since the file was introduced.
Worth flagging as a side effect: the bundled `examples/qa_agent_evaluation/eval/eval-report.md` reports results (Faithfulness 78.2%, Answer Relevancy 100.0%, 2 test cases), so those numbers did not come from this code path as committed. Not suggesting anything is wrong with the numbers themselves — more that anyone trying to reproduce the report from the repo as-is will not get there.
### Reproduce
```
curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
&& sed -n '/def create_test_case_from_trace/,/return test_case/p' \
awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval/metrics.py
```
Confirmed at 472fbc6.
### Suggested fix
Remove the `exit()`, and probably demote the three `print()` calls to `logging.debug` while you are in there.
Happy to open a PR if that is useful.
Contributor guide
Research direction
Start in examples/qa_agent_evaluation/eval/metrics.py at create_test_case_from_trace(), where the unconditional exit() follows the three debug prints. Run the bundled evaluation through run_evaluation.py and confirm it reaches LLMTestCase construction and completes instead of terminating on the first trace; consider the suggested logging adjustment if in scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100