Auto-trigger eval runs when new data points are added to a dataset

Summary

Allow users to configure an existing dataset so that, whenever new data points (rows) are added to it, a selected set of evals automatically runs against just those new rows. Today evals are a point-in-time snapshot — anything appended after a run is un-evaluated until the user remembers to re-run, which is wasteful (re-run on the whole dataset) or stale (skip it).

Motivation

• Continuous regression monitoring — append production traces into a regression dataset and have quality scores stay current
• Lower cost — evaluate only the delta, not the whole dataset
• Faster feedback loops — see eval scores on new rows without manual re-runs

Current state (verified in code)

The good news: most of the infrastructure already exists. The gap is the wiring + a small data-model addition.

What exists:

• Dataset insert paths are async and centralized:
  • futureagi/tracer/views/dataset.py — add_to_existing_dataset() and add_to_new_dataset()
  • futureagi/tracer/tasks/dataset.py — process_spans_chunk_task() does the actual Row / Cell bulk_create
• Eval execution runs on Temporal:
  • futureagi/tfc/temporal/evaluations/workflows.py — RunEvaluationWorkflow and RunEvaluationBatchWorkflow (with built-in max_concurrent controls)
  • futureagi/tfc/temporal/evaluations/client.py — start_evaluation_workflow() / start_evaluation_batch_workflow() are the entry points
• Eval result model + status enum: futureagi/model_hub/models/evaluation.py
• Soft-delete cascade signal pattern (template for adding new signals): futureagi/model_hub/signals.py:141-208

What's missing:

1. No post_save signal on Row — there's no event fired when a row is appended. Today the only Dataset-related signal is the soft-delete cascade.
2. No dataset ↔ eval linkage model — EvalTemplate is workspace-scoped; nothing today says "these evals belong to this dataset and should auto-run."
3. Single-row eval execution doesn't exist — run_evaluation() (futureagi/tracer/views/custom_eval_config.py:282-365) filters spans by observation type across a project version. There is no entry point that takes a list of Row IDs.
4. Row → eval-input bridge missing — Evaluation.input_data is a JSONField populated from span attributes today; mapping a Row's cells onto the eval template's expected inputs is the trickiest piece of this feature.

Proposed design

New model: DatasetEvalConfig

• dataset (FK to Dataset)
• eval_template (FK to EvalTemplate) — multiple configs per dataset → multiple evals per row
• enabled (bool)
• max_concurrent (int, default 5) — propagated to the Temporal batch workflow
• debounce_seconds (int, default 30) — coalesce bulk inserts into one workflow start
• column_mapping (JSON) — explicit mapping from dataset column names to the eval template's expected input fields. This makes the Row → eval-input bridge user-controlled rather than guessed.

Trigger flow

1. After process_spans_chunk_task finishes its bulk_create, emit a rows_appended(dataset_id, row_ids) event.
2. A handler looks up DatasetEvalConfig rows where dataset=dataset_id, enabled=True.
3. For each config, the handler debounces (coalesces row IDs over debounce_seconds) and calls start_evaluation_batch_workflow(eval_template_id, row_ids, column_mapping).
4. A new Temporal activity, run_eval_on_row_activity, builds the eval input from the row + mapping and invokes the existing single-eval runner.

Why a new event instead of post_save directly: the actual insert is in bulk_create, which doesn't fire post_save per row. The cleanest hook is at the end of process_spans_chunk_task where row IDs are already known. A simple Django signal there is enough — no need for a real event bus.

Frontend (separate slice)

• New "Auto-eval" tab in the dataset detail view: toggle, multi-select for evals, column mapping editor
• Status badge on the dataset card: "Auto-eval: on (3 evals)"
• Per-row eval scores rendered inline (uses existing eval-result UI)

The frontend can ship as a follow-up issue once the backend is in place — it doesn't have to land in the same PR.

Acceptance criteria

Backend:

• DatasetEvalConfig model + migration + admin
• CRUD API: create / list / update / delete configs per dataset
• Hook in process_spans_chunk_task emits row-appended event with new row IDs
• Handler debounces and starts a Temporal batch workflow per enabled config
• New run_eval_on_row_activity builds eval input via column_mapping
• Failures on individual rows do not block other rows or future inserts
• Per-workspace rate limit on auto-eval workflows (re-uses existing concurrency caps)
• Insert latency unchanged (auto-eval scheduling is fully async after bulk_create)

Visibility / ops:

• Evaluation rows produced by auto-eval carry the source DatasetEvalConfig ID for provenance
• Metrics for: auto-eval triggers fired, rows evaluated, attempts failed
• Disabling a config halts new triggers but does not cancel in-flight workflows

Out of scope

• Re-evaluating existing rows when an eval definition changes (separate feature)
• Streaming auto-eval scores to external systems (webhooks)
• Per-row threshold gating that blocks downstream pipelines (we just record the score)
• Frontend implementation — tracked as a follow-up issue once the backend lands

Start by reading futureagi/tracer/tasks/dataset.py, futureagi/tracer/views/dataset.py, and the signal pattern in futureagi/model_hub/signals.py:141-208, then trace the Temporal entry points in futureagi/tfc/temporal/evaluations/. The work is complete when the model, CRUD API, asynchronous row-trigger flow, mapped evaluation activity, provenance, metrics, rate limits, and failure isolation meet the listed backend and operations acceptance criteria.