ai-forever / ai-forever/harness-bench-fast
Result: opencode CLI + GLM 5.2 (z.ai, agent1st_v14.1) — 388/391 (99.2%)
- Dominant language
- Python
- Stars
- 51
- Forks
- 11
- Avg merge
- 4m
- Merged PRs (30d)
- 1
Description
## Result row for README
| Harness | Model | Result | % | Steps | Tokens |
| --- | --- | ---: | ---: | ---: | ---: |
| opencode CLI | GLM 5.2 (z.ai, agent1st_v14.1-glm) | 388/391 | 99.2% | — | — |
## Details
- **Harness:** `opencode` CLI (standalone `opencode run` subprocess, each task fresh)
- **Model:** GLM 5.2 (z.ai coding plan, `zai-coding-plan/glm-5.2`)
- **Agent:** `A/agent1st_glm` (Agent1st Protocol **v14.1** for GLM 5.2, 662 lines)
- **Config:** `opencode-bench.json` (LSP off, MCP off), `--dangerously-skip-permissions`
- **Reasoning variant:** `high` (non-max effort — faster, fewer 900s timeouts)
- **Concurrency:** 1
- **Per-task timeout:** 900s
- **OpenCode version:** 1.18.11
- **max_tokens:** 131072 (`OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX`, CAS-134)
## Per-wave breakdown
| Wave | Tasks | Passed | % |
| --- | ---: | ---: | ---: |
| core (1–30) | 30 | 30 | 100.0% |
| extra (31–60) | 30 | 30 | 100.0% |
| more (61–100) | 40 | 40 | 100.0% |
| hard (101–150) | 50 | 50 | 100.0% |
| extreme (151–205) | 55 | 54 | 98.2% |
| diagnostic (206–221) | 16 | 16 | 100.0% |
| memory (222–253) | 32 | 31 | 96.9% |
| agentic (254–298) | 45 | 45 | 100.0% |
| VCS (299–313) | 15 | 15 | 100.0% |
| skills (314–333) | 20 | 20 | 100.0% |
| adversarial (334–354) | 21 | 21 | 100.0% |
| tbench-lite (355–374) | 20 | 20 | 100.0% |
| CLI (375–391) | 17 | 16 | 94.1% |
| **Total** | **391** | **388** | **99.2%** |
## Failures (3)
| Task | Reason |
| --- | --- |
| `task_202_zip_sales_consolidation` | `region_totals.csv`: float `630.0` instead of int `630` — numeric type precision in CSV output |
| `task_223_memory_city_timezone_script` | `MEMORY.md` missing `- Город: Москва` line — memory persistence gap (script written, memory not updated) |
| `task_378_cli_cfgctl_layered_merge` | `effective.env` missing — bespoke CLI layered merge not completed |
All three are deterministic, single-attempt failures — not systematic. Two of three (202, 378) are format/type precision edge cases; one (223) is memory persistence.
## Evidence
Full per-task result JSON: `jobs/results-glm52-v016.json` (391 tasks, 388 passed, pass_rate = 0.992)
```json
{"total": 391, "passed": 388, "pass_rate": 0.992}
```
## Reproduction
```bash
cd /path/to/harness-bench-fast
export OPENCODE_CONFIG=opencode-bench.json
uv run python -m harness_bench run-cli \
--cli-command 'opencode run -m zai-coding-plan/glm-5.2 --agent A/agent1st_glm --variant high --dangerously-skip-permissions' \
--concurrency 1 \
--timeout 900 \
--json-output jobs/results-glm52-v016.json \
--allow-task-failures --keep
```
## Notes
- Agent protocol: [Agent1st Protocol v14.1](https://agents.md) — v14.1 = v14 + practical patch from a live session audit (subagent gate enforcement status, assumption-check diagnostic command, drift-recovery narrowing, post-compaction cascade guard, quote-anchor cost control, worker-role section).
- Same harness + agent stack, same 391-task set: **GLM 5.2 v14.1 = 99.2%**, on par with Qwen 3.8 Max v5.3 (99.2%), ahead of DeepSeek V4 Flash v37.3 (98.2%).
[results-glm52-v016.json](https://github.com/user-attachments/files/30638433/results-glm52-v016.json)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.