get2knowio / get2knowio/maverick
Agent output evaluation framework
- Dominant language
- Python
- Stars
- 4
- Forks
- 0
- Avg merge
- 17h 37m
- Merged PRs (30d)
- 7
Description
## Summary
Introduce a lightweight agent output evaluation framework that enables structured quality assessment of agent outputs beyond binary success/fail, supporting regression detection and quality tracking across workflow runs.
## Motivation
Maverick currently evaluates agent execution in binary terms: `AgentResult.success` is `True` or `False`. This is sufficient for workflow control flow but insufficient for tracking **quality** over time. As Maverick's agent fleet grows and backends diversify, we need to detect:
- Quality regressions when upgrading models or prompts
- Behavioral differences between backends (Claude vs. Copilot)
- Drift in agent output patterns over time
- Whether prompt engineering changes actually improve outputs
### Inspiration from Oracle's WayFlow
Oracle's [WayFlow](https://oracle.github.io/wayflow/25.4.2/) includes explicit evaluation capabilities:
- "Evaluate WayFlow Assistants" — structured evaluation of assistant behavior
- "Evaluate Assistant Conversations" — conversation-level quality assessment
While WayFlow's evaluation is general-purpose (any assistant), Maverick can build a more targeted framework focused on software development task quality.
## Proposed Architecture
### Module structure
```
src/maverick/
├── evaluation/
│ ├── __init__.py
│ ├── protocol.py # Evaluator Protocol
│ ├── evaluators/
│ │ ├── __init__.py
│ │ ├── code_review.py # Evaluates review agent outputs
│ │ ├── implementation.py # Evaluates implementer outputs
│ │ └── generation.py # Evaluates generator outputs (PR desc, commit msg)
│ ├── metrics.py # Quality metric definitions
│ ├── report.py # Evaluation report generation
│ └── registry.py # Evaluator registry
```
### Evaluator Protocol
```python
from __future__ import annotations
from typing import Protocol
from dataclasses import dataclass, field
@dataclass(frozen=True)
class QualityMetric:
"""A single quality measurement."""
name: str
score: float # 0.0 to 1.0
weight: float = 1.0 # Relative importance
details: str = "" # Human-readable explanation
@dataclass(frozen=True)
class EvaluationResult:
"""Aggregated evaluation of an agent's output."""
agent_name: str
step_name: str
metrics: tuple[QualityMetric, ...]
overall_score: float # Weighted average, 0.0 to 1.0
passed: bool # Whether it meets minimum threshold
timestamp: str
metadata: dict[str, Any] = field(default_factory=dict)
class AgentEvaluator(Protocol):
"""Evaluates the quality of an agent's output."""
@property
def agent_type(self) -> str:
"""Which agent type this evaluator handles."""
...
async def evaluate(
self,
*,
agent_result: AgentResult,
context: dict[str, Any],
expected: dict[str, Any] | None = None,
) -> EvaluationResult:
...
```
### Example evaluators
**Code Review Evaluator** — assesses review agent output quality:
- **Coverage**: Did the review address all changed files?
- **Specificity**: Are findings tied to specific lines/functions, not vague?
- **Actionability**: Can each finding be acted on without further clarification?
- **False positive rate**: Are findings valid (when ground truth is available)?
- **Severity calibration**: Are severity levels appropriate?
**Implementation Evaluator** — assesses implementer agent output quality:
- **Completeness**: Were all requested tasks addressed?
- **Test coverage**: Did the implementation include tests?
- **Convention adherence**: Does generated code follow project conventions?
- **Build health**: Does the result compile/lint/typecheck?
**Generation Evaluator** — assesses text generation quality:
- **Format compliance**: Does the output match the expected format (e.g., PR description template)?
- **Content coverage**: Are all required sections present?
- **Length appropriateness**: Not too terse, not verbose
### Integration points
1. **Post-step evaluation**: After an agent step completes, optionally run its evaluator
2. **Event emission**: Emit evaluation results as `ProgressEvent` for TUI display
3. **Session log**: Append evaluation results to the session journal
4. **CLI reporting**: `maverick evaluate` command for ad-hoc evaluation of past runs
5. **Threshold gates**: Optional `min_quality: 0.7` on agent steps that fails the step if quality is too low
### DSL integration (optional, future)
```yaml
- name: review
type: agent
agent: code_reviewer
evaluate: true # Run evaluator after completion
min_quality: 0.7 # Fail step if quality below threshold
```
## Scope
### In scope (this issue)
- `AgentEvaluator` Protocol definition
- `EvaluationResult` and `QualityMetric` data models
- Evaluator registry
- At least one concrete evaluator (code review or generation)
- Integration with event system
### Out of scope (future work)
- Historical quality tracking / persistence
- Dashboard / visualization
- Automated prompt optimization based on evaluation results
- Cross-backend comparison tooling
## Acceptance Criteria
- [ ] `AgentEvaluator` Protocol defined
- [ ] `EvaluationResult` and `QualityMetric` frozen dataclasses
- [ ] Evaluator registry with register/get/list operations
- [ ] At least one concrete evaluator with meaningful metrics
- [ ] Event type for evaluation results
- [ ] Tests for protocol compliance and evaluator logic
- [ ] Documentation of how to create custom evaluators
## References
- Oracle WayFlow evaluation guides: https://oracle.github.io/wayflow/25.4.2/core/howtoguides/index.html
- Current `AgentResult`: `src/maverick/agents/result.py`
- Current event system: `src/maverick/dsl/events.py`
Contributor guide
Research direction
Start by reading src/maverick/agents/result.py and src/maverick/dsl/events.py to understand the existing result and event models. Then review the proposed evaluation module structure and acceptance criteria, including the registry and one concrete evaluator. Done means the protocol and frozen data models exist, an evaluator can be registered and run, evaluation events are integrated, tests cover the behavior, and custom evaluator usage is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100