ash-project / ash-project/evals

Feat: Implement `:tagged_summary` report format

Open
#2 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Elixir
Stars
49
Forks
6
PR merge metrics
No merged PRs in 30d

Description

### Is your feature request related to a problem? Please describe.

**Labels:** `feature`, `reporting`, `formatter`

**Dependencies:** Blocks Ticket #1 (YAML Metadata Parsing)

**Description:**
As a model evaluator, I want a report format that groups evaluation results by their tags. This will allow for a more granular analysis of a model's performance, helping to identify strengths and weaknesses in specific areas like `otp`, `ecto`, or `pattern-matching`.

This requires creating a new report format option, `:tagged_summary`.

### Describe the solution you'd like

**Acceptance Criteria:**
* The `Evals.Formatter.format_report/3` function accepts a new value for the `:format` option: `:tagged_summary`.
* When `format: :tagged_summary` is used, the report output should:
1. List each unique tag found across all completed evals.
2. For each tag, display the average score of all evals that include that tag.
3. The list of tags should be sorted alphabetically for consistent output.
4. The final report must be clean and human-readable.
* The implementation must not break the existing `:full` and `:summary` report formats.
* If no evals have tags, this section of the report should be gracefully omitted.

**Implementation Notes:**
* This work will be done in `evals/formatter.ex`.
* A new private function, such as `format_tagged_summary(results)`, will likely be needed.
* The logic will involve:
1. Reducing the `results` list into a map where keys are tags and values are lists of scores (e.g., `%{otp: [1.0, 0.5], elixir_core: [1.0, 1.0, 0.5], ...}`).
2. Transforming this map to calculate the average score for each tag.
3. Sorting the results by tag name before formatting them into strings.

**Example Output:**
```
... (Overall Summary) ...

TAGGED SUMMARY:
----------------------------------------
elixir-core | 83.3%
ecosystem | 100.0%
otp | 75.0%
pattern-matching | 100.0%
security | 0.0%

... (Detailed Results or end of report) ...
```

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in evals/formatter.ex at Evals.Formatter.format_report/3 and compare the existing :full and :summary formats. Implement the :tagged_summary option using completed eval results, then verify unique tags are alphabetically sorted, averages are human-readable, untagged results are omitted gracefully, and existing formats remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
analytics
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.