Enable Running Evaluations in Agent Playground

Summary

Enable running the platform's 70+ built-in evaluations (accuracy, toxicity, relevance, faithfulness, etc.) directly within the Agent Playground. This should support two modes:

1. Node-level evaluation — An Evaluation Node that runs inline as part of the workflow, evaluating data from a single upstream node.
2. Agent-level evaluation — Evaluate the complete agent's behavior by collecting inputs from multiple nodes across the graph and running evaluations on the aggregated results.

Motivation

Users currently evaluate agent outputs after-the-fact by exporting results and running evaluations separately. This disconnect means:

• Issues are caught too late in the development cycle
• There's no way to use evaluation results to influence workflow behavior (e.g., retry on low quality)
• Users can't assess end-to-end agent quality without leaving the playground

Bringing evaluations into the playground enables:

• Inline quality gates: Check if an LLM response is toxic before sending it to the user
• End-to-end agent scoring: Evaluate the full agent output considering context from multiple nodes (e.g., compare final answer against original input + retrieved context)
• Iterative improvement: Run the agent, see evaluation scores, tweak prompts, repeat — all in one place

Requirements

Node-Level Evaluation:

• An Evaluation Node that can be placed inline in the workflow
• Users select an evaluation template from the platform's 70+ built-in options
• The node takes an input (data to evaluate) and an optional reference (ground truth / context)
• Outputs:
  • Evaluation result (score, pass/fail, details)
  • Passthrough of original input (so the eval doesn't break the data flow)
• Users can set a pass/fail threshold
• Configurable fail action: continue, stop execution, or route to a fallback
• Score should be displayed on the node after execution (color-coded pass/fail)

Agent-Level Evaluation:

• An "Evaluate Agent" panel/button in the execution results view
• Users can select one or more evaluation templates to run
• Users can map outputs from ANY node in the graph to evaluation fields:
  • e.g., "Input" = Node3.response, "Reference" = Node1.query, "Context" = Node2.retrieved_docs
• This allows evaluating the complete agent behavior, not just individual steps
• Support running multiple evaluations simultaneously (e.g., accuracy + toxicity + relevance all at once)
• Results dashboard showing:
  • Per-evaluation scores with pass/fail indicators
  • Aggregate score across all evaluations
  • Historical trend across multiple runs
• Evaluation configurations should be saveable and reusable across runs

Acceptance Criteria

Node-level:

• Users can add an Evaluation node and select from available evaluation templates
• Users can set a pass/fail threshold
• During execution, the node runs the selected evaluation on its input
• Evaluation score is displayed on the node (color-coded pass/fail)
• The passthrough output forwards original data so evals don't break the chain
• Users can configure what happens on failure (continue, stop, or route to fallback)

Agent-level:

• Users can run evaluations on the complete agent after execution
• Users can map outputs from multiple nodes as evaluation inputs
• Multiple evaluations can run on the same execution simultaneously
• Evaluation results are displayed in a summary dashboard
• Evaluation configurations can be saved and reused across runs

Start at the Agent Playground entry point and execution results view, then trace how evaluation templates and outputs are currently represented. Define the node-level and agent-level data mappings, thresholds, failure actions, and reusable configurations before implementation. Done means both evaluation modes meet all listed acceptance criteria, including displayed scores and historical results.