awslabs / awslabs/agent-evaluation

Fails to run on windows

Open
#110 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
372
Forks
51
PR merge metrics
No merged PRs in 30d

Description

If you run this tool on windows you get a couple of errors with the default config.
First
```
File "C:\Users\test\.venv\Lib\site-packages\agenteval\evaluators\canonical\evaluator.py", line 70, in __init__
"system": jinja_env.get_template(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\.venv\Lib\site-packages\jinja2\environment.py", line 1016, in get_template
return self._load_template(name, globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\.venv\Lib\site-packages\jinja2\environment.py", line 975, in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\.venv\Lib\site-packages\jinja2\loaders.py", line 126, in load
source, filename, uptodate = self.get_source(environment, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\.venv\Lib\site-packages\jinja2\loaders.py", line 373, in get_source
posixpath.join(self._template_root, *split_template_path(template))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\.venv\Lib\site-packages\jinja2\loaders.py", line 36, in split_template_path
raise TemplateNotFound(template)
```

If I edit the loaders.py file in jinja to this it works
```
def split_template_path(template: str) -> t.List[str]:
"""Split a path into segments and perform a sanity check. If it detects
'..' in the path it will raise a `TemplateNotFound` error.
"""
pieces = []
template = template.replace("\\", "/") # this is the new line

```
After that it fails with
```
File "C:\Users\test\.venv\Lib\site-packages\agenteval\summary.py", line 53, in _write_summary
f.write(summary)
File "C:\Users\Kbechtel\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]

```
Which I can fix by editing the summary.py file
```
def _write_summary(path: str, summary: str):
with open(path, "w+", encoding='utf-8') as f: # adding the encoding here.
f.write(summary)

```
Heres my evaluator and target for reference:
```
evaluator:
model: claude-3
aws_profile: arandomprofile
aws_region: us-east-1
target:
type: bedrock-agent
bedrock_agent_id: nope
bedrock_agent_alias_id: nope
aws_profile: arandomprofile
aws_region: us-east-1
bedrock_session_attributes:
first_name: user-name
bedrock_prompt_session_attributes:
timezone: user-timezone
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the default configuration on Windows and inspect agenteval/evaluators/canonical/evaluator.py around line 70 and agenteval/summary.py around line 53. Compare template loading and summary writing behavior across platforms; done means the default evaluation runs without TemplateNotFound or encoding errors on Windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.