Proposal: EvalPort adapter for portable AgentKernelArena results (task_result.yaml <-> TestCase/Result)

Hi maintainers — really nice project; the A/B harness plus a centralized, agent-independent evaluator.py/score.py is exactly the kind of setup that benefits from a portable results format, so I wanted to float an idea rather than send a surprise PR.

What EvalPort is

EvalPort is an Apache-2.0 open interchange standard for portable LLM/agent evaluation: a small set of typed objects (TestCase, Grader, Result, ResultSet) plus a Python/TS SDK (evalport-sdk), with 37 real adapter packages already merged for eval/observability tools (e.g. lm-eval-harness-openeval-adapter, mlflow-openeval-adapter). The point isn't to replace anyone's own format — it's a translation layer so results from different harnesses can be compared, archived, or fed into other tooling without bespoke glue for each source.

Why AgentKernelArena maps cleanly

I read through src/testcases.py, src/evaluator.py, and src/score.py to check this against real fields rather than guessing:

• TestCaseResult(test_case_id, shape, execution_time_ms, metadata) from src/testcases.py maps almost 1:1 onto EvalPort's TestCase (one per task/shape, with metadata carrying things like benchmark_method).
• The per-task config.yaml contract (compile_command, correctness_command, performance_command, task_type) is naturally an EvalPort Grader — compilation/correctness/performance are exactly the three graded dimensions src/score.py::score() already scores independently (20 / 100 / speedup_ratio*100).
• task_result.yaml, as written by write_task_result() in src/evaluator.py (pass_compilation, pass_correctness, base_execution_time, best_optimized_execution_time, speedup_ratio, score, baseline_benchmark_methods, optimized_benchmark_methods), maps onto an EvalPort Result; a full run directory's aggregated results map onto a ResultSet — which is what src/tools/compare_runs.py already does by hand for two runs. A ResultSet would let that same comparison work across N runs, or against results from a different harness entirely.

Concrete sketch (illustrative, using your real field names)

from evalport_sdk import TestCase, Result, ResultSet
import yaml

def task_result_to_evalport(workspace, task_config):
    data = yaml.safe_load((workspace / "task_result.yaml").read_text())

    case = TestCase(
        id=data["task_name"],
        metadata={
            "task_type": task_config.get("task_type"),
            "target_kernel_functions": task_config.get("target_kernel_functions"),
        },
    )
    result = Result(
        test_case_id=case.id,
        passed=data["pass_compilation"] and data["pass_correctness"],
        score=data.get("score"),
        metrics={
            "pass_compilation": data["pass_compilation"],
            "pass_correctness": data["pass_correctness"],
            "base_execution_time_ms": data["base_execution_time"],
            "best_optimized_execution_time_ms": data["best_optimized_execution_time"],
            "speedup_ratio": data["speedup_ratio"],
        },
        error=data.get("compilation_error_message") or data.get("correctness_error_message"),
    )
    return case, result

This would live as a new agentkernelarena-openeval-adapter package in EvalPort's adapters/ — no changes needed to this repo's own task_result.yaml/scoring pipeline, and it wouldn't touch src/module_registration.py or the agent-integration surface at all. It's a pure downstream reader.

What I'm asking

Would a small, additive adapter of this kind (living entirely in the EvalPort repo, read-only against task_result.yaml/config.yaml) be useful to you, or does something like this already exist / conflict with planned work (I saw the visualization dashboard and compare_runs.py already cover part of this)? If there's interest I'm happy to build and maintain it there, and to open a PR here only for the minimal doc pointer if that's welcome — not proposing any change to your scoring or evaluation code itself.

— Sahi, independent contributor (not affiliated with AMD)

Start by reading src/testcases.py, src/evaluator.py, src/score.py, and src/tools/compare_runs.py, then verify the task_result.yaml and config.yaml fields described in the proposal. Done means maintainers decide whether an external EvalPort adapter is useful, whether existing work conflicts, and whether a documentation pointer in this repository is wanted.