Mapping Harbor task/result files to EvalPort's TestCase/ResultSet (no code changes needed)

Hi — I'm Sahi, building EvalPort, an open interchange format for portable LLM/agent eval data (TestCase/Grader/ResultSet/GraderResult, JSON Schema + Python+TS SDKs). Came across ccbench through the Harbor task format and wanted to flag a pretty clean mapping in case it's useful — this is just an observation, not asking for anything to be merged here.

Looked at a real task dir (tasks/grep-backreferences-cpp-caribou-380) plus result.json/config.json from a run, and scripts/success_by_group.py's reading of them. Since everything's already plain JSON/TOML/Markdown on disk, this doesn't depend on what language the harness itself is written in — it's a file-format boundary, not an SDK integration:

• Each task dir → one EvalPort TestCase: id = task_name, input = instruction.md's content (already agent-facing prose), context = the relevant features/*.md stage docs, and solution/code/ (the oracle reference) hangs off metadata since it's a whole codebase, not a string.
• tests/test.sh + Harbor's verifier → one EvalPort Grader of type custom (exec-based, not text/similarity) — verifier_result.rewards.reward is a float (1.0 = pass, per success_by_group.py), which maps straight onto GraderResult.score (0.0–1.0) with passed: score == 1.0.
• A job run → one ResultSet: results[].test_case_id = task_name, grader_results[0].score/passed from verifier_result.rewards.reward, duration_ms from agent_execution.started_at/finished_at, and config.json's agent.name/model_name → runner.name/provider.model.

Sketch:

{
  "id": "grep-backreferences-cpp-caribou-380",
  "input": "# Build your own grep - C++\n- Implement: #sb5 Single Backreference, #tg1 Multiple Backreferences, #xe5 Nested Backreferences\n...",
  "context": [
    "environment/code/features/26-sb5.md",
    "environment/code/features/27-tg1.md",
    "environment/code/features/28-xe5.md"
  ],
  "graders": [
    { "id": "gr_harbor_verifier", "type": "custom", "params": { "handler": "harbor:test.sh" } }
  ],
  "metadata": { "ccbench.solution_ref": "solution/code/" }
}

{
  "test_case_id": "grep-backreferences-cpp-caribou-380",
  "grader_results": [
    { "grader_id": "gr_harbor_verifier", "type": "custom", "score": 1.0, "passed": true }
  ],
  "passed": true,
  "duration_ms": 842000
}

Precedent for this kind of thing: ragas-openeval-adapter reads Ragas's public EvaluationResult from the outside and emits/reads EvalPort — no changes needed in Ragas itself. A harbor-openeval-adapter could work the same way here: walk jobs/<run>/**/result.json, join against config.json, emit a ResultSet. That'd give ccbench results a portable format other harnesses/dashboards can read without writing a ccbench-specific parser, and would work for anyone else on the Harbor task format too, not just this repo.

Happy to sketch a working adapter against a real jobs.tar.gz sample if that'd be useful — otherwise just flagging in case the mapping is handy.

— Sahi, independent contributor (not affiliated with this project)

Start with tasks/grep-backreferences-cpp-caribou-380, its instruction.md, features/*.md, result.json, config.json, and scripts/success_by_group.py. If pursued, done would be a concrete adapter that walks jobs//**/result.json, joins config.json, and emits the described ResultSet; the issue currently requests no repository change.