ash-project / ash-project/evals

Feat: Add new eval type 'match_output' for stdout grading

Open
#3 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Elixir
Stars
49
Forks
6
PR merge metrics
No merged PRs in 30d

Description

### Is your feature request related to a problem? Please describe.

**Labels:** `feature`, `eval-engine`, `grader`

**Description:**
As an eval author, I want to create evaluations that test the standard output (`stdout`) of a script. This is crucial for testing command-line behavior, simple script outputs, or any task where the primary result is printed to the console rather than returned from a function.

This involves creating a new eval `type` named `match_output` and implementing the corresponding grading logic.

### Describe the solution you'd like

**Acceptance Criteria:**
* The framework supports a new eval `type: "match_output"` in the YAML definition.
* The YAML for this type must support a new key: `eval.expected_output` (string).
* The `Evals.from_yml/1` function must correctly parse this new structure.
* A new function clause must be added to `Evals.grade/3` to handle evals where `type` is `:match_output`.
* This new grader should:
1. Execute the model-generated code using `Evals.write_and_eval/3`.
2. Capture the `stdout` from the execution.
3. Compare the captured `stdout` with the `expected_output` from the eval's definition.
4. The comparison should be robust against leading/trailing whitespace on both the actual and expected output.
5. A successful match results in a grade of `1`; a mismatch results in `0`.

**Implementation Notes:**
* The main logic will be a new function head for `Evals.grade/3` in `evals.ex`.
* The `Evals.write_and_eval/3` function already captures `stdout`, so it can be used directly.
* Use `String.trim/1` on both the captured output and the expected output before comparison.

**Example YAML:**
```yaml
type: match_output
messages:
- type: user
text: "Write a script that prints 'Hello, World!' to the console."
eval:
expected_output: |
Hello, World!
```

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in evals.ex by tracing Evals.from_yml/1, Evals.grade/3, and Evals.write_and_eval/3 to understand the existing eval flow and stdout capture. Implement the new YAML structure and grading path so trimmed expected and captured output produce 1 on a match and 0 otherwise, then verify both parsing and grading behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
ai, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.