get2knowio / get2knowio/maverick

Agent output evaluation framework

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
4
Forks
0
Avg merge
17h 37m
Merged PRs (30d)
7

Description

## Summary

Introduce a lightweight agent output evaluation framework that enables structured quality assessment of agent outputs beyond binary success/fail, supporting regression detection and quality tracking across workflow runs.

## Motivation

Maverick currently evaluates agent execution in binary terms: `AgentResult.success` is `True` or `False`. This is sufficient for workflow control flow but insufficient for tracking **quality** over time. As Maverick's agent fleet grows and backends diversify, we need to detect:

- Quality regressions when upgrading models or prompts
- Behavioral differences between backends (Claude vs. Copilot)
- Drift in agent output patterns over time
- Whether prompt engineering changes actually improve outputs

### Inspiration from Oracle's WayFlow

Oracle's [WayFlow](https://oracle.github.io/wayflow/25.4.2/) includes explicit evaluation capabilities:

- "Evaluate WayFlow Assistants" — structured evaluation of assistant behavior
- "Evaluate Assistant Conversations" — conversation-level quality assessment

While WayFlow's evaluation is general-purpose (any assistant), Maverick can build a more targeted framework focused on software development task quality.

## Proposed Architecture

### Module structure

```
src/maverick/
├── evaluation/
│ ├── __init__.py
│ ├── protocol.py # Evaluator Protocol
│ ├── evaluators/
│ │ ├── __init__.py
│ │ ├── code_review.py # Evaluates review agent outputs
│ │ ├── implementation.py # Evaluates implementer outputs
│ │ └── generation.py # Evaluates generator outputs (PR desc, commit msg)
│ ├── metrics.py # Quality metric definitions
│ ├── report.py # Evaluation report generation
│ └── registry.py # Evaluator registry
```

### Evaluator Protocol

```python
from __future__ import annotations

from typing import Protocol
from dataclasses import dataclass, field

@dataclass(frozen=True)
class QualityMetric:
"""A single quality measurement."""
name: str
score: float # 0.0 to 1.0
weight: float = 1.0 # Relative importance
details: str = "" # Human-readable explanation

@dataclass(frozen=True)
class EvaluationResult:
"""Aggregated evaluation of an agent's output."""
agent_name: str
step_name: str
metrics: tuple[QualityMetric, ...]
overall_score: float # Weighted average, 0.0 to 1.0
passed: bool # Whether it meets minimum threshold
timestamp: str
metadata: dict[str, Any] = field(default_factory=dict)

class AgentEvaluator(Protocol):
"""Evaluates the quality of an agent's output."""

@property
def agent_type(self) -> str:
"""Which agent type this evaluator handles."""
...

async def evaluate(
self,
*,
agent_result: AgentResult,
context: dict[str, Any],
expected: dict[str, Any] | None = None,
) -> EvaluationResult:
...
```

### Example evaluators

**Code Review Evaluator** — assesses review agent output quality:
- **Coverage**: Did the review address all changed files?
- **Specificity**: Are findings tied to specific lines/functions, not vague?
- **Actionability**: Can each finding be acted on without further clarification?
- **False positive rate**: Are findings valid (when ground truth is available)?
- **Severity calibration**: Are severity levels appropriate?

**Implementation Evaluator** — assesses implementer agent output quality:
- **Completeness**: Were all requested tasks addressed?
- **Test coverage**: Did the implementation include tests?
- **Convention adherence**: Does generated code follow project conventions?
- **Build health**: Does the result compile/lint/typecheck?

**Generation Evaluator** — assesses text generation quality:
- **Format compliance**: Does the output match the expected format (e.g., PR description template)?
- **Content coverage**: Are all required sections present?
- **Length appropriateness**: Not too terse, not verbose

### Integration points

1. **Post-step evaluation**: After an agent step completes, optionally run its evaluator
2. **Event emission**: Emit evaluation results as `ProgressEvent` for TUI display
3. **Session log**: Append evaluation results to the session journal
4. **CLI reporting**: `maverick evaluate` command for ad-hoc evaluation of past runs
5. **Threshold gates**: Optional `min_quality: 0.7` on agent steps that fails the step if quality is too low

### DSL integration (optional, future)

```yaml
- name: review
type: agent
agent: code_reviewer
evaluate: true # Run evaluator after completion
min_quality: 0.7 # Fail step if quality below threshold
```

## Scope

### In scope (this issue)
- `AgentEvaluator` Protocol definition
- `EvaluationResult` and `QualityMetric` data models
- Evaluator registry
- At least one concrete evaluator (code review or generation)
- Integration with event system

### Out of scope (future work)
- Historical quality tracking / persistence
- Dashboard / visualization
- Automated prompt optimization based on evaluation results
- Cross-backend comparison tooling

## Acceptance Criteria

- [ ] `AgentEvaluator` Protocol defined
- [ ] `EvaluationResult` and `QualityMetric` frozen dataclasses
- [ ] Evaluator registry with register/get/list operations
- [ ] At least one concrete evaluator with meaningful metrics
- [ ] Event type for evaluation results
- [ ] Tests for protocol compliance and evaluator logic
- [ ] Documentation of how to create custom evaluators

## References

- Oracle WayFlow evaluation guides: https://oracle.github.io/wayflow/25.4.2/core/howtoguides/index.html
- Current `AgentResult`: `src/maverick/agents/result.py`
- Current event system: `src/maverick/dsl/events.py`

Contributor guide

Open the contributing guide

Research direction

Start by reading src/maverick/agents/result.py and src/maverick/dsl/events.py to understand the existing result and event models. Then review the proposed evaluation module structure and acceptance criteria, including the registry and one concrete evaluator. Done means the protocol and frozen data models exist, an evaluator can be registered and run, evaluation events are integrated, tests cover the behavior, and custom evaluator usage is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.