ai-forever / ai-forever/harness-bench-fast

Result: opencode CLI + GLM 5.2 (z.ai, agent1st_v14.1) — 388/391 (99.2%)

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
51
Forks
11
Avg merge
4m
Merged PRs (30d)
1

Description

## Result row for README

| Harness | Model | Result | % | Steps | Tokens |
| --- | --- | ---: | ---: | ---: | ---: |
| opencode CLI | GLM 5.2 (z.ai, agent1st_v14.1-glm) | 388/391 | 99.2% | — | — |

## Details

- **Harness:** `opencode` CLI (standalone `opencode run` subprocess, each task fresh)
- **Model:** GLM 5.2 (z.ai coding plan, `zai-coding-plan/glm-5.2`)
- **Agent:** `A/agent1st_glm` (Agent1st Protocol **v14.1** for GLM 5.2, 662 lines)
- **Config:** `opencode-bench.json` (LSP off, MCP off), `--dangerously-skip-permissions`
- **Reasoning variant:** `high` (non-max effort — faster, fewer 900s timeouts)
- **Concurrency:** 1
- **Per-task timeout:** 900s
- **OpenCode version:** 1.18.11
- **max_tokens:** 131072 (`OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX`, CAS-134)

## Per-wave breakdown

| Wave | Tasks | Passed | % |
| --- | ---: | ---: | ---: |
| core (1–30) | 30 | 30 | 100.0% |
| extra (31–60) | 30 | 30 | 100.0% |
| more (61–100) | 40 | 40 | 100.0% |
| hard (101–150) | 50 | 50 | 100.0% |
| extreme (151–205) | 55 | 54 | 98.2% |
| diagnostic (206–221) | 16 | 16 | 100.0% |
| memory (222–253) | 32 | 31 | 96.9% |
| agentic (254–298) | 45 | 45 | 100.0% |
| VCS (299–313) | 15 | 15 | 100.0% |
| skills (314–333) | 20 | 20 | 100.0% |
| adversarial (334–354) | 21 | 21 | 100.0% |
| tbench-lite (355–374) | 20 | 20 | 100.0% |
| CLI (375–391) | 17 | 16 | 94.1% |
| **Total** | **391** | **388** | **99.2%** |

## Failures (3)

| Task | Reason |
| --- | --- |
| `task_202_zip_sales_consolidation` | `region_totals.csv`: float `630.0` instead of int `630` — numeric type precision in CSV output |
| `task_223_memory_city_timezone_script` | `MEMORY.md` missing `- Город: Москва` line — memory persistence gap (script written, memory not updated) |
| `task_378_cli_cfgctl_layered_merge` | `effective.env` missing — bespoke CLI layered merge not completed |

All three are deterministic, single-attempt failures — not systematic. Two of three (202, 378) are format/type precision edge cases; one (223) is memory persistence.

## Evidence

Full per-task result JSON: `jobs/results-glm52-v016.json` (391 tasks, 388 passed, pass_rate = 0.992)

```json
{"total": 391, "passed": 388, "pass_rate": 0.992}
```

## Reproduction

```bash
cd /path/to/harness-bench-fast
export OPENCODE_CONFIG=opencode-bench.json

uv run python -m harness_bench run-cli \
--cli-command 'opencode run -m zai-coding-plan/glm-5.2 --agent A/agent1st_glm --variant high --dangerously-skip-permissions' \
--concurrency 1 \
--timeout 900 \
--json-output jobs/results-glm52-v016.json \
--allow-task-failures --keep
```

## Notes

- Agent protocol: [Agent1st Protocol v14.1](https://agents.md) — v14.1 = v14 + practical patch from a live session audit (subagent gate enforcement status, assumption-check diagnostic command, drift-recovery narrowing, post-compaction cascade guard, quote-anchor cost control, worker-role section).
- Same harness + agent stack, same 391-task set: **GLM 5.2 v14.1 = 99.2%**, on par with Qwen 3.8 Max v5.3 (99.2%), ahead of DeepSeek V4 Flash v37.3 (98.2%).

[results-glm52-v016.json](https://github.com/user-attachments/files/30638433/results-glm52-v016.json)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.