monad-developers / monad-developers/ultrafuzz
Support provider-agnostic coding harnesses via tight Smithers adapters
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 18
- Avg merge
- 11h 10m
- Merged PRs (30d)
- 194
Description
Umbrella for provider-agnostic harness support, rewritten around the tight-adapter ruling.
Governing constraint
Ultrafuzz carries tight adapters over Smithers' built-in harness registry and never re-implements Smithers internals. Smithers ships 14 built-in CLI harnesses and already exposes every binding axis as constructor options: harness = the agent class, provider = baseURL + apiKey, model = an opaque provider catalogue ID, protocol = the class. smithers agents capabilities --format json publishes each harness's binary, emittedFlags, optionMappings and resume kind; smithers agents doctor and smithers agents test already validate reachability.
A tight adapter is one ~76-line file in packages/runtime/src/templates/smithers/agents/, modelled on claude.tsx, that reads only auth / api_key_env / config_dir from its own [agents.<Ref>Agent] table, maps them onto existing Smithers options, wraps child env in workflowControlChildEnvironment and credentials in workflowControlCredentialValue, and does nothing else.
What this umbrella no longer includes
- No new configuration schema. No
[providers.*], no[harnesses.*], no independentharness/providerfields, noProviderBinding/HarnessCapabilities/QualifiedHarnessBindingtypes, no binding validator.models.*.agentis already an open agent-ref schema and[agents.*]is already a generic record, so a new harness needs zero config work. (#658 closed; PR #682 rejected.) - No ultrafuzz-side capability contract. Capability truth is
smithers agents capabilities/doctor. Ultrafuzz consumes it; it does not define a second one. - No G1-G10 conformance suite. Qualification for a pairing is: one real provider smoke run on a pinned CLI, plus two unit assertions (credential absent from argv, credential present in child env) and one filesystem assertion (no harness state outside the run-scoped roots).
Sequencing (children are now independent)
- #659 — Pi + OpenRouter tight adapter
- #660 — correct the DeepSeek pairing terminology (docs/comments only)
- #661 — DeepSeek Harness (
dsh): not a tight adapter — no Smithers registry entry exists. Owner decision required; recommended resolution is upstream. - #662 — OpenCode + OpenRouter tight adapter with run-scoped state
- #663 — Modal image: pin the new harness packages and extend the existing command prober
- #664 — closed; residual doc wording folded into #660
Codex remains the shipped default until a qualified alternative beats it on real-provider evidence. That is a default-selection decision, not an architecture claim.
Adding a harness — the whole checklist
- Confirm the harness is in
smithers agents capabilities; recordpackageExport,binary,optionMappings,resume. If it is absent, stop — this is not an adapter task. - Copy
claude.tsx, change five things: the Smithers class import, the[agents.<Ref>Agent]table name, the default auth mode, the defaultapi_key_env, and the unattended-execution posture. - Register in
agents/index.tsx(agentFactories) and inAGENT_TEMPLATESinpackages/runtime/src/init.ts. - Map ref -> binary in
AGENT_EXECUTABLESinpackages/runtime/src/doctor.ts, usingsurface.binary. - Add tests next to the existing ones: index export,
agentFactoriesentry, no module-level construction, and a generated-adapter load test asserting the env passed throughworkflowControlChildEnvironment. - Touch
packages/configonly for a hard constraint the harness genuinely rejects. Never add per-harness env allow-list entries to the runtime layer.
Review gate: under ~100 lines, no fs beyond readFileSync, no fetch, no argv indexing, no usage arithmetic, no session handling, no hardcoded model names or context sizes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Treat this as an umbrella rather than a single implementation task: the independent work is tracked in #659, #660, #661, #662 and #663. For a specific adapter, first inspect smithers agents capabilities and packages/runtime/src/templates/smithers/agents/claude.tsx, then follow the listed registration, doctor mapping and test locations. Done means the selected child meets the tight-adapter review gate and its stated qualification checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, devtools, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100