awslabs / awslabs/Agent-EvalKit

metrics.py: unconditional exit() in create_test_case_from_trace prevents test case creation

Open Beginner friendly
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Summary

In `examples/qa_agent_evaluation/eval/metrics.py`, `create_test_case_from_trace()` calls `exit()` unconditionally. It sits directly after the three debug `print()` calls and **before** both the `if not query or not answer:` guard and the `LLMTestCase(...)` construction:

```python
print(f" Extracted query: {query}")
print(f" Extracted answer: {answer}")
print(f" Extracted search results: {search_results}")
exit()

if not query or not answer:
...
test_case = LLMTestCase(input=query, actual_output=answer, retrieval_context=search_results)
return test_case
```

The function can therefore never return a test case — it terminates the interpreter on the first trace it processes, so `run_evaluation.py` cannot complete against the bundled example.

### Since when

`metrics.py` has a single commit in its path history: 151fb44 (`add qa agent evaluation example`, 2025-11-24). The `exit()` has been present since the file was introduced.

Worth flagging as a side effect: the bundled `examples/qa_agent_evaluation/eval/eval-report.md` reports results (Faithfulness 78.2%, Answer Relevancy 100.0%, 2 test cases), so those numbers did not come from this code path as committed. Not suggesting anything is wrong with the numbers themselves — more that anyone trying to reproduce the report from the repo as-is will not get there.

### Reproduce

```
curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
&& sed -n '/def create_test_case_from_trace/,/return test_case/p' \
awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval/metrics.py
```

Confirmed at 472fbc6.

### Suggested fix

Remove the `exit()`, and probably demote the three `print()` calls to `logging.debug` while you are in there.

Happy to open a PR if that is useful.

Contributor guide

Open the contributing guide

Research direction

Start in examples/qa_agent_evaluation/eval/metrics.py at create_test_case_from_trace(), where the unconditional exit() follows the three debug prints. Run the bundled evaluation through run_evaluation.py and confirm it reaches LLMTestCase construction and completes instead of terminating on the first trace; consider the suggested logging adjustment if in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.