Add lineage-aware Agent Eval dataset generation from Intake ATIF traces

Summary

Add a Platform-owned, lineage-aware path for materializing Agent Eval datasets from normalized Intake ATIF traces, and for composing already-materialized per-agent datasets into a shared catalog dataset.

This is needed by TestCrew Studio, but the capability is intentionally generic: Platform owns trace discovery, immutable dataset materialization, Fileset storage, grader bindings, and lineage. TestCrew owns the agent/catalog selection UX, family/access semantics, and dataset review experience.

A working implementation is available on:

• Branch: codex/testcrew-studio-integration/puhuim
• Commit: 2c57ff5e4f349920fada5998be8f86ba1f8c2d37

Problem

Intake persists ATIF/ATOF execution evidence, while Evaluator and Data Designer consume reviewable datasets/tasks. Before this change, clients had to implement two orchestration gaps outside Platform:

1. Select traces for one agent and convert them into a stable Agent Eval dataset.
2. Combine multiple ready per-agent datasets into a shared catalog-level dataset.

Keeping this logic in each client causes inconsistent extraction, loss of grader/reference data, and incomplete lineage across Trace → Agent Dataset → Catalog Dataset.

Raw ATIF traces and datasets must remain distinct:

• A Trace is immutable execution evidence and retains the original trajectory.
• A Dataset row is a normalized, reviewable evaluation case derived from a trace. It contains an input, observed output, optional reference, grader bindings/results, and provenance.
• A Catalog Dataset is a union of already-materialized Agent Datasets, not a union of raw traces.

Proposed ownership

Platform

• Filter normalized Intake traces by exact agent identity.
• Materialize a selected set of traces into an immutable Dataset Fileset.
• Compose existing Dataset Filesets using a versioned row contract.
• Preserve grader bindings, historical grader results, and end-to-end lineage.
• Store the reviewable payload in canonical Parquet for Data Designer and Agent Eval reuse.

Client / TestCrew Studio

• Present agents and their selectable traces.
• Manage QA family/catalog concepts and authorization.
• Select all or a subset of traces and submit Platform jobs.
• Select ready Agent Datasets for catalog composition.
• Provide the Dataset Review UX for input, output, reference, grader, and lineage.

API design

1. Discover traces for an agent

GET /apis/intake/v2/workspaces/{workspace}/traces?filter={"agent_name":"tea"}&mode=preview

agent_name is exposed as a first-class trace field and filter. Existing durable spans remain the source of truth; the ClickHouse trace index migration rebuilds the index from those spans so existing ATIF does not need to be re-ingested.

2. Build an Agent Dataset from selected traces

POST /apis/data-designer/v2/workspaces/{workspace}/jobs/build-dataset
Content-Type: application/json

{
  "name": "tea-traces-dataset-build",
  "spec": {
    "destination": {
      "name": "tea-traces-dataset-v1",
      "description": "Selected workflow traces for the tea agent"
    },
    "source": {
      "kind": "intake-traces",
      "agent_name": "tea",
      "trace_ids": ["trace-1", "trace-2"],
      "grader_refs": ["default/testcrew-atif-quality-v2"]
    }
  }
}

The job validates that every selected root trace belongs to the requested agent.

3. Compose Agent Datasets into a shared Catalog Dataset

{
  "name": "qa-catalog-dataset-build",
  "spec": {
    "destination": {
      "name": "qa-catalog-dataset-v1"
    },
    "source": {
      "kind": "datasets",
      "datasets": [
        "default/lta-traces-dataset-v1",
        "default/tea-traces-dataset-v1"
      ]
    }
  }
}

Composition accepts only Dataset Filesets using the versioned contract. Identical stable row IDs are deduplicated; conflicting content for the same ID fails the job instead of silently choosing a value.

Dataset contract

The output is an immutable Fileset with:

• purpose=dataset
• dataset.contract=nemo.agent-eval-dataset.v1
• data/data.parquet
• manifest.json
• typed dataset metadata containing record_count, grader_refs, and dataset-level lineage

Each review-oriented row contains:

• id
• instruction
• observed_output
• reference
• grader_refs
• grader_results
• agent_name
• trace_id
• session_id
• evaluation_id
• test_case_id
• trace_status
• trace_started_at
• lineage

Complex reference, grader, and provenance values are represented as canonical JSON strings. This permits heterogeneous agent-specific oracles in one Parquet schema and allows the artifact to be used directly as a Data Designer seed dataset.

When normalized root input/output is incomplete, extraction recovers the first user input and final assistant output from the raw ATIF/ATOF evidence without discarding the original trace lineage.

Implemented changes

• Add agent_name to the Intake trace domain/API/index.
• Add a ClickHouse trace-index migration that preserves and reindexes existing spans.
• Add backward-compatible Dataset Fileset metadata for data path, row count, grader references, and lineage.
• Add the Data Designer build-dataset NemoJob.
• Support kind=intake-traces for per-agent dataset generation.
• Support kind=datasets for catalog-level composition.
• Preserve trace, dataset, metric, evaluation, test-case, and session provenance.
• Generate and update Platform and plugin OpenAPI specifications.
• Document both API flows.
• Add unit/API/repository/migration coverage.

Validation

• 87 targeted tests passed.
• Ruff passed for all touched paths.
• Type checking passed for all touched paths.
• Local live Platform E2E built two per-agent datasets and composed them successfully.
• A QA Astra deployment preserved all pre-existing Platform resources during upgrade.
• Live validation materialized:
  • LTA Agent Dataset: 67 rows
  • tea Agent Dataset: 4 rows
  • QA Catalog Dataset: 71 unique rows
• The final Parquet artifact contained the expected lta=67 and tea=4 distribution, and every row retained an instruction, trace ID, grader binding, and both intake-trace and dataset lineage.

Remaining work / review questions

• Review and merge the implementation branch.
• Confirm the long-term public naming/location of the dataset materialization API.
• Decide whether Platform should later expose a native Dataset Review UI; the current integration keeps review UX in TestCrew.
• Decide whether a first-class Catalog resource is needed. The current Platform primitive is intentionally a generic Dataset composition operation.
• Evaluate pagination/streaming for very large trace selections and catalog unions.
• Address deployment-level CPU job scheduling constraints independently from this API. The current QA deployment serializes these jobs because its RWO Files PVC is pinned to one node.

Non-goals

• Copying or replacing the raw ATIF trajectory.
• Adding TestCrew-specific QA Family or catalog authorization concepts to Platform.
• Automatically running Data Designer enrichment or Agent Eval after dataset creation.
• Changing metric semantics or promoting agent candidates.

Start by reviewing branch codex/testcrew-studio-integration/puhuim at commit 2c57ff5e4f349920fada5998be8f86ba1f8c2d37, then trace the Intake v2 traces endpoint and Data Designer build-dataset job. Run the 87 targeted tests and review the OpenAPI and migration coverage; completion means resolving the naming, review, pagination, and deployment questions and merging the implementation.