Proposal: an EvalPort adapter for PawBench's tasks + submissions (interchange format for portable eval data)

## Proposal: an `evalport` adapter for PawBench's tasks + submissions

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an Apache-2.0 interchange format (JSON Schema + Python/TS SDK, `evalport-sdk`) for portable LLM eval test cases, graders, and results. I read through the actual task/grading/submission code in this repo before writing this (not just the README), and I think PawBench's data model is a genuinely good fit — enough that I'd like to build and contribute the adapter myself if that's welcome.

### What I actually looked at

- `data/pawbench-v1.0/tasks/T001_claweval_M005_score_canon.md` (and neighboring task files) — the YAML front-matter + Prompt/Expected Behavior/Grading Criteria/Automated Checks/LLM Judge Rubric structure
- `pawbench/grader.py` — `grade_task()`, the three `grading_type`s (`automated`/`llm_judge`/`hybrid`), `GradeResult` (`task_id`, `score`, `max_score`, `grading_type`, `breakdown`, `notes`, `score_simple`), and `_combine_grades()`'s weighted combination
- `submissions/pawbench-4models-opusjudge-20260529__qwen3.6-plus__qwenpaw.json` — a real committed submission (`run`, `model`, `harness`, `overall`, `automated`, `judge`, `tasks`, `tasks_errored`, `tasks_missing`, plus the `by_complexity`/`by_environment`/`by_scenario`/`by_scenario_top`/`by_modality`/`by_channel`/`by_capability`/`by_source`/`by_category`/`by_subcategory`/`by_grading` slice breakdowns)

### The mapping

**Task → `TestCase`**

| PawBench (real fields, `T001_claweval_M005_score_canon.md`) | EvalPort `TestCase` |
|---|---|
| front-matter `id: M005_score_canon` | `id` |
| `## Prompt` body | `input` |
| `## Expected Behavior` body | `expected_output` |
| `labels.capabilities`, `labels.modality.type/.channels`, `labels.scenario`, `labels.complexity`, `labels.environment`, plus the submission's `by_source`/`by_category` keys | `tags` (e.g. `["capability:Tool_Use", "scenario:Content_Creation/Design", "complexity:L3", "environment:closed", "modality:multimodal", "source:claweval"]`) |
| `timeout_seconds: 1200` | `timeout_ms` (×1000) |

**Grading → `graders[]`, using the real `grading_weights`**

`T001` has `grading_type: hybrid` with `grading_weights: {automated: 0.3, llm_judge: 0.7}`. That maps onto two `Grader` entries on the same `TestCase`, weighted the same way EvalPort's own `weight` field already supports:

```python
graders = [
{
"id": "T001_automated",
"type": "code",
"weight": 0.3,
"params": {"language": "python", "source": automated_checks_source}, # the grade() body, verbatim
},
{
"id": "T001_llm_judge",
"type": "llm_judge",
"weight": 0.7,
"params": {"model": "claude-opus-4-5-20251101", "prompt": llm_judge_rubric_text}, # real default from grader.py
},
]
```

One honest wrinkle I want to flag rather than paper over: PawBench's `grade()` function signature is `grade(transcript: list, workspace_path: str) -> dict` — it scores the *whole agent transcript plus final workspace filesystem state*, not a single `actual_output` string. EvalPort's `code` grader type doesn't currently pin down a calling convention for `params.source` beyond "language + source," so this isn't a hard blocker, but it's worth resolving explicitly (e.g. does the adapter serialize workspace file contents into `TestCase.context`, or is this a case for a `pawbench.transcript`/`pawbench.workspace_snapshot` `metadata` extension?) before I write code that guesses.

**Submission JSON → `ResultSet`**

The committed `submissions/*.json` files are already *aggregated* (one JSON per model×harness run, not one row per task) — there's no raw per-task `GradeResult` dump in the repo, only the rolled-up `overall`/`by_*` breakdowns. So a faithful adapter has to be honest about what's actually round-trippable:

- `run` + `model` + `harness` → `ResultSet.run_id` (e.g. `"qwen3.6-plus__qwenpaw"`), `suite_id: "pawbench-v1.0"`
- `overall`/`automated`/`judge` → `ResultSet.summary.avg_score` plus `summary.metadata["pawbench.automated"]` / `["pawbench.judge"]` (EvalPort's built-in `summary.by_grader` is keyed by grader id, not by PawBench's five-dimension taxonomy, so the `by_scenario`/`by_capability`/`by_complexity`/`by_modality`/`by_environment`/`by_source`/`by_category`/`by_subcategory` blocks would live under `ResultSet.metadata["pawbench.by_scenario"]` etc. — preserved verbatim, not recomputed, the same `.` metadata convention this repo's other adapters already use)
- `tasks_errored`/`tasks_missing` → `summary.failed`/`summary.skipped` (best-effort; PawBench doesn't distinguish per-task which specific tasks errored vs. went missing in the aggregate file, so per-`Result` fidelity is limited to what's actually in the file)

### Comparable adapters already in the repo (verified, not asserted from memory)

- [`financebench-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/financebench-openeval-adapter) — closest shape match: a fixed benchmark with real questions + gold answers + an `llm_judge` grader (because FinanceBench's free-text answers don't fit `exact_match` either) + real per-model result files with an existing human-annotated label, carried into EvalPort's `Result`/`GraderResult` verbatim rather than re-graded.
- [`lm-eval-harness-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/lm-eval-harness-openeval-adapter) — comparable on the "harness" side: documents two real discrepancies it found between `lm-eval`'s docstrings and its actually-installed behavior, and is explicit about what does/doesn't round-trip (it can't reconstruct a live `lm_eval.api.task.Task`, same category of limitation I'm flagging above for PawBench's transcript+workspace grader signature).

### What I'd propose

If this is a direction the maintainers are open to, I'd build `pawbench-openeval-adapter` under `adapters/` in the EvalPort repo (not a PR against PawBench itself — no code changes needed here), following the same pattern: `to_openeval()`/`from_openeval()` for tasks, `result_to_openeval()` for submissions, real fixtures pulled from files actually in this repo, tests run against the real `openeval.validate.validate_suite()`/`validate_result_set()`. I'd rather get a maintainer's read on the grader-signature question above (and on whether `by_scenario`-style slice metadata under a `pawbench.` namespace is a reasonable round-trip target, versus something you'd want expressed differently) before writing it, rather than build first and ask later.

Happy to close this out if PawBench↔EvalPort interop isn't something you want to carry, no worries either way.

— Sahi, independent contributor (not affiliated with agentscope-ai)