redhat-et / redhat-et/docsclaw
feat: add evaluation harness for agent behavior testing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 9m
- Merged PRs (30d)
- 1
Description
Summary
Add a test harness that validates agent behavior through repeatable scenarios — not just code correctness (unit tests) but whether the agent produces useful, accurate output for given inputs.
Motivation
Agent quality depends on prompt engineering, skill design, and tool orchestration — none of which are covered by go test. Frameworks like Google ADK emphasize evaluation as a first-class concern. DocsClaw should have its own eval framework without depending on external frameworks.
Proposed design
An eval scenario is a YAML file describing input, expected behavior, and pass criteria:
name: executive-summary
description: Agent should produce a structured summary from a report
config-dir: testdata/executive-assistant
input: |
Summarize this Q1 report: Revenue $4.2M, up 12%.
Key driver: Enterprise segment grew 25%.
Risk: Two large renewals pending in Q2.
expect:
contains:
- "revenue"
- "enterprise"
- "risk"
max_length: 500
has_sections:
- "Summary"
- "Key decisions"
A docsclaw eval subcommand runs scenarios against a live or mocked LLM:
docsclaw eval scenarios/ # run all scenarios
docsclaw eval scenarios/hr/ # run HR-specific scenarios
Key features
- Deterministic mode: fixed seed / temperature 0 for reproducibility
- Golden file comparison: save a known-good output, diff against it
- Scoring rubric: LLM-as-judge for subjective quality (optional)
- CI integration: exit code reflects pass/fail for pipeline gating
- Cost tracking: report token usage per scenario
Related
- Closes gap identified in #33 (ADK evaluation)
- Scenarios in
docs/demo/could become the first eval cases
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing scenarios in docs/demo/ and the evaluation gap described in #33. Then trace how DocsClaw exposes CLI subcommands and agent execution. Done means a documented eval framework can run YAML scenarios, report pass/fail and costs, and support the proposed reproducibility and CI behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai, cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100