redhat-et / redhat-et/docsclaw

feat: add evaluation harness for agent behavior testing

Open
#35 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2
Forks
1
Avg merge
6h 9m
Merged PRs (30d)
1

Description

Summary

Add a test harness that validates agent behavior through repeatable scenarios — not just code correctness (unit tests) but whether the agent produces useful, accurate output for given inputs.

Motivation

Agent quality depends on prompt engineering, skill design, and tool orchestration — none of which are covered by go test. Frameworks like Google ADK emphasize evaluation as a first-class concern. DocsClaw should have its own eval framework without depending on external frameworks.

Proposed design

An eval scenario is a YAML file describing input, expected behavior, and pass criteria:

name: executive-summary
description: Agent should produce a structured summary from a report
config-dir: testdata/executive-assistant
input: |
  Summarize this Q1 report: Revenue $4.2M, up 12%.
  Key driver: Enterprise segment grew 25%.
  Risk: Two large renewals pending in Q2.
expect:
  contains:
    - "revenue"
    - "enterprise"
    - "risk"
  max_length: 500
  has_sections:
    - "Summary"
    - "Key decisions"

A docsclaw eval subcommand runs scenarios against a live or mocked LLM:

docsclaw eval scenarios/       # run all scenarios
docsclaw eval scenarios/hr/    # run HR-specific scenarios

Key features

  • Deterministic mode: fixed seed / temperature 0 for reproducibility
  • Golden file comparison: save a known-good output, diff against it
  • Scoring rubric: LLM-as-judge for subjective quality (optional)
  • CI integration: exit code reflects pass/fail for pipeline gating
  • Cost tracking: report token usage per scenario

Related

  • Closes gap identified in #33 (ADK evaluation)
  • Scenarios in docs/demo/ could become the first eval cases

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing scenarios in docs/demo/ and the evaluation gap described in #33. Then trace how DocsClaw exposes CLI subcommands and agent execution. Done means a documented eval framework can run YAML scenarios, report pass/fail and costs, and support the proposed reproducibility and CI behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.