bytechefhq / bytechefhq/bytechef

[feature] AI Agent Evaluations

Open
#4,553 0 comments 0 reactions 1 assignee Claimed by @ivicac View on GitHub
ai backend enhancement feature-flag frontend
Dominant language
Java
Stars
1k
Forks
170
Avg merge
11h 25m
Merged PRs (30d)
115

Description

## Summary

Add an automated evaluation system for ByteChef AI agents with test scenarios, LLM/deterministic judges, async run execution, and a dedicated UI panel — integrated with `spring-ai-community/agent-judge`.

## Key Features

- **Test Scenarios**: Single-turn (message → response) and multi-turn (simulated conversation) scenarios
- **Two-Level Judges**: Agent-level judges (run on all scenarios) + scenario-level judges (scoped)
- **Judge Types**: LLM rule-based + deterministic (contains text, regex, response length, JSON schema, similarity)
- **Async Execution**: Runs execute asynchronously with progress tracking and cancellation
- **Results & History**: Score tracking, judge verdicts with explanations, conversation transcript storage
- **UI Panel**: New "Evals" tab in AI Agent Editor with Tests/Judges/Runs sub-tabs

## Design Spec

`docs/superpowers/specs/2026-03-15-agent-evaluations-design.md`

## Implementation Plan

`docs/superpowers/plans/2026-03-15-agent-evaluations.md`

## Phase 1 Scope

- Agent-level evaluations only (workflow-level deferred)
- Sequential scenario execution (parallel deferred)
- Informational results only (save gating deferred)
- TOOL_USAGE judge deferred (requires structured tool event capture)

## Tech Stack

- Backend: Spring Boot 4, Spring Data JDBC, Spring AI, spring-ai-community/agent-judge
- Frontend: React 19, TypeScript, Zustand, TanStack Query, GraphQL

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.