ash-project / ash-project/evals
Feat: Add new eval type 'match_output' for stdout grading
- Dominant language
- Elixir
- Stars
- 49
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
### Is your feature request related to a problem? Please describe.
**Labels:** `feature`, `eval-engine`, `grader`
**Description:**
As an eval author, I want to create evaluations that test the standard output (`stdout`) of a script. This is crucial for testing command-line behavior, simple script outputs, or any task where the primary result is printed to the console rather than returned from a function.
This involves creating a new eval `type` named `match_output` and implementing the corresponding grading logic.
### Describe the solution you'd like
**Acceptance Criteria:**
* The framework supports a new eval `type: "match_output"` in the YAML definition.
* The YAML for this type must support a new key: `eval.expected_output` (string).
* The `Evals.from_yml/1` function must correctly parse this new structure.
* A new function clause must be added to `Evals.grade/3` to handle evals where `type` is `:match_output`.
* This new grader should:
1. Execute the model-generated code using `Evals.write_and_eval/3`.
2. Capture the `stdout` from the execution.
3. Compare the captured `stdout` with the `expected_output` from the eval's definition.
4. The comparison should be robust against leading/trailing whitespace on both the actual and expected output.
5. A successful match results in a grade of `1`; a mismatch results in `0`.
**Implementation Notes:**
* The main logic will be a new function head for `Evals.grade/3` in `evals.ex`.
* The `Evals.write_and_eval/3` function already captures `stdout`, so it can be used directly.
* Use `String.trim/1` on both the captured output and the expected output before comparison.
**Example YAML:**
```yaml
type: match_output
messages:
- type: user
text: "Write a script that prints 'Hello, World!' to the console."
eval:
expected_output: |
Hello, World!
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start in evals.ex by tracing Evals.from_yml/1, Evals.grade/3, and Evals.write_and_eval/3 to understand the existing eval flow and stdout capture. Implement the new YAML structure and grading path so trimmed expected and captured output produce 1 on a match and 0 otherwise, then verify both parsing and grading behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100