bytechefhq / bytechefhq/bytechef
[feature] AI Agent Evaluations
- Dominant language
- Java
- Stars
- 1k
- Forks
- 170
- Avg merge
- 11h 25m
- Merged PRs (30d)
- 115
Description
## Summary
Add an automated evaluation system for ByteChef AI agents with test scenarios, LLM/deterministic judges, async run execution, and a dedicated UI panel — integrated with `spring-ai-community/agent-judge`.
## Key Features
- **Test Scenarios**: Single-turn (message → response) and multi-turn (simulated conversation) scenarios
- **Two-Level Judges**: Agent-level judges (run on all scenarios) + scenario-level judges (scoped)
- **Judge Types**: LLM rule-based + deterministic (contains text, regex, response length, JSON schema, similarity)
- **Async Execution**: Runs execute asynchronously with progress tracking and cancellation
- **Results & History**: Score tracking, judge verdicts with explanations, conversation transcript storage
- **UI Panel**: New "Evals" tab in AI Agent Editor with Tests/Judges/Runs sub-tabs
## Design Spec
`docs/superpowers/specs/2026-03-15-agent-evaluations-design.md`
## Implementation Plan
`docs/superpowers/plans/2026-03-15-agent-evaluations.md`
## Phase 1 Scope
- Agent-level evaluations only (workflow-level deferred)
- Sequential scenario execution (parallel deferred)
- Informational results only (save gating deferred)
- TOOL_USAGE judge deferred (requires structured tool event capture)
## Tech Stack
- Backend: Spring Boot 4, Spring Data JDBC, Spring AI, spring-ai-community/agent-judge
- Frontend: React 19, TypeScript, Zustand, TanStack Query, GraphQL
Contributor guide
Assessment
This issue has not been assessed yet.