ash-project / ash-project/evals
Feat: Add support for eval metadata keys in YAML parser
- Dominant language
- Elixir
- Stars
- 49
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
### Is your feature request related to a problem? Please describe.
**Labels:** `enhancement`, `parser`, `good-first-issue`
**Description:**
As an eval author, I want to include structured metadata in my `.yml` files to better describe and categorize each evaluation. This data will be foundational for future features like advanced filtering and reporting.
Currently, our parser only extracts the core functional keys. We need to extend it to parse additional descriptive keys: `id`, `description`, `difficulty`, and `tags`.
### Describe the solution you'd like
**Acceptance Criteria:**
* The `Evals.from_yml/1` function is updated to parse the following new optional keys from the root of a YAML eval file:
* `id` (string)
* `description` (string)
* `difficulty` (string, e.g., "easy", "medium", "hard")
* `tags` (list of strings)
* The parsed keys should be converted to atoms (e.g., `:id`, `:difficulty`).
* The parsed values should be stored in the resulting `eval` map that is passed through the evaluation pipeline.
* The parser must remain backward-compatible and **not** fail if these new keys are absent.
* The `difficulty` string should be converted to an atom (e.g., `"easy"` -> `:easy`).
**Implementation Notes:**
* This work will primarily be in `evals.ex`, within the `from_yml/1` function.
* The existing `remap_key/4` helper can be used for most of this.
* For `difficulty`, a custom mapping function might be needed to convert the string to an atom.
```elixir
# Example of how from_yml should transform the data
# from_yml(%{"id" => "...", "difficulty" => "easy", ...})
# -> %{id: "...", difficulty: :easy, ...}
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start in evals.ex at the Evals.from_yml/1 function and inspect the existing remap_key/4 helper. Confirm how optional root keys are currently handled, then verify that id, description, difficulty, and tags are stored with the required atom keys and values without breaking files that omit them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100