Proposal: map TaskDefinition/TaskRunResult/BenchmarkResult onto EvalPort's TestCase/Result/ResultSet

I've been reading through clawbench/schemas.py and the trace-based scoring pipeline (completion/trajectory/behavior/judge axes, pass^k reliability, the 13-mode failure taxonomy) and think ClawBench's result data maps cleanly onto EvalPort — an open interchange format for portable eval datasets/results (JSON schema + a Python openeval SDK + a TS SDK). Wanted to check the mapping against the real field names before proposing anything, rather than show up with a guess.

Mapping, based on sdk/python/openeval/types.py and clawbench/schemas.py as they actually are today:

• TaskDefinition → EvalPort TestCase: id → id; simulated-user turns (SessionPhase.user.turns[].message via TaskDefinition.normalized_phases()) → input (list of turn strings); completion.execution_checks → code-type Graders (command + expected_exit_code as params); judge.rubric → an llm_judge Grader with params.rubric; tier/family/scenario/capabilities/etc. → metadata.

• TaskRunResult → EvalPort Result: task_id → test_case_id; transcript.assistant_text → actual_output; duration_ms → duration_ms; delivery_outcome.value == "pass" → passed; error → error. The four sub-results (completion_result, trajectory_result, behavior_result, judge_result) each become their own GraderResult (grader_id = gr_completion/gr_trajectory/gr_behavior/gr_judge, type="custom", score, passed = score >= pass_threshold) — the same pattern adapters/ragas-openeval-adapter in the EvalPort repo already uses to turn each independent metric into its own grader instead of collapsing everything into one number.

• BenchmarkResult → EvalPort ResultSet: submission_id → run_id; {model, provider} → provider; timestamp → started_at; task_results/tier_results/scenario_results → summary.

Sketch — this would live as a standalone clawbench-openeval-adapter package (same playbook as the existing ragas-openeval-adapter/autogen-openeval-adapter: works against ClawBench's public Pydantic models from the outside, nothing needs to merge into ClawBench core):

from openeval.types import TestCase, Grader, Result, GraderResult, ResultSet
from clawbench.schemas import TaskDefinition, TaskRunResult

def task_to_testcase(task: TaskDefinition) -> TestCase:
    turns = [t.message for phase in task.normalized_phases() for t in phase.user.turns]
    graders: list[Grader] = [
        Grader(
            id=f"gr_exec_{c.name}",
            type="code",
            params={"command": c.command, "expected_exit_code": c.expected_exit_code},
        )
        for c in task.completion.execution_checks
    ]
    if task.judge:
        graders.append(Grader(id="gr_judge", type="llm_judge", params={"rubric": task.judge.rubric}))
    return TestCase(
        id=task.id,
        input=turns or [""],
        graders=graders,
        metadata={
            "tier": task.tier.value,
            "family": task.family.value,
            "scenario": task.scenario.value if task.scenario else "",
        },
    )

def run_to_result(run: TaskRunResult) -> Result:
    def gr(name: str, sub, score: float) -> GraderResult:
        return GraderResult(
            grader_id=f"gr_{name}", type="custom", score=score,
            passed=score >= 0.7, reason=getattr(sub, "reason", ""),
        )
    return Result(
        test_case_id=run.task_id,
        passed=run.delivery_outcome.value == "pass",
        grader_results=[
            gr("completion", run.completion_result, run.completion_result.score),
            gr("trajectory", run.trajectory_result, run.trajectory_result.score),
            gr("behavior", run.behavior_result, run.behavior_result.score),
            gr("judge", run.judge_result, run.judge_result.score),
        ],
        actual_output=run.transcript.assistant_text,
        duration_ms=run.duration_ms,
        error={"message": run.error} if run.error else None,
    )

Happy to build this out as an adapter if it's useful. Two things I'd rather ask than assume: (1) is ResultSet.summary the right place for the tier/scenario rollups, or should those stay ClawBench-specific and only per-run Results cross over? (2) Partner Trace Spec already defines a JSONL interchange for traces — is EvalPort's Result/ResultSet meant to be complementary to that (scores/verdicts) rather than overlapping with it (raw trace), or would this proposal be redundant with where PARTNER_TRACE_SPEC.md is headed?

— Sahi, independent contributor (not affiliated with this project)

Start by reading clawbench/schemas.py, sdk/python/openeval/types.py, and PARTNER_TRACE_SPEC.md, then inspect the existing ragas-openeval-adapter and autogen-openeval-adapter patterns. Confirm the proposed field mappings and package boundary with maintainers; done means a standalone adapter with validated TaskDefinition, TaskRunResult, and BenchmarkResult conversions.