monad-developers / monad-developers/ultrafuzz

Support provider-agnostic coding harnesses via tight Smithers adapters

Open
#653 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deferred-post-v0.1 enhancement
Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Umbrella for provider-agnostic harness support, rewritten around the tight-adapter ruling.

Governing constraint

Ultrafuzz carries tight adapters over Smithers' built-in harness registry and never re-implements Smithers internals. Smithers ships 14 built-in CLI harnesses and already exposes every binding axis as constructor options: harness = the agent class, provider = baseURL + apiKey, model = an opaque provider catalogue ID, protocol = the class. smithers agents capabilities --format json publishes each harness's binary, emittedFlags, optionMappings and resume kind; smithers agents doctor and smithers agents test already validate reachability.

A tight adapter is one ~76-line file in packages/runtime/src/templates/smithers/agents/, modelled on claude.tsx, that reads only auth / api_key_env / config_dir from its own [agents.<Ref>Agent] table, maps them onto existing Smithers options, wraps child env in workflowControlChildEnvironment and credentials in workflowControlCredentialValue, and does nothing else.

What this umbrella no longer includes

  • No new configuration schema. No [providers.*], no [harnesses.*], no independent harness / provider fields, no ProviderBinding / HarnessCapabilities / QualifiedHarnessBinding types, no binding validator. models.*.agent is already an open agent-ref schema and [agents.*] is already a generic record, so a new harness needs zero config work. (#658 closed; PR #682 rejected.)
  • No ultrafuzz-side capability contract. Capability truth is smithers agents capabilities / doctor. Ultrafuzz consumes it; it does not define a second one.
  • No G1-G10 conformance suite. Qualification for a pairing is: one real provider smoke run on a pinned CLI, plus two unit assertions (credential absent from argv, credential present in child env) and one filesystem assertion (no harness state outside the run-scoped roots).

Sequencing (children are now independent)

  • #659 — Pi + OpenRouter tight adapter
  • #660 — correct the DeepSeek pairing terminology (docs/comments only)
  • #661 — DeepSeek Harness (dsh): not a tight adapter — no Smithers registry entry exists. Owner decision required; recommended resolution is upstream.
  • #662 — OpenCode + OpenRouter tight adapter with run-scoped state
  • #663 — Modal image: pin the new harness packages and extend the existing command prober
  • #664 — closed; residual doc wording folded into #660

Codex remains the shipped default until a qualified alternative beats it on real-provider evidence. That is a default-selection decision, not an architecture claim.

Adding a harness — the whole checklist

  1. Confirm the harness is in smithers agents capabilities; record packageExport, binary, optionMappings, resume. If it is absent, stop — this is not an adapter task.
  2. Copy claude.tsx, change five things: the Smithers class import, the [agents.<Ref>Agent] table name, the default auth mode, the default api_key_env, and the unattended-execution posture.
  3. Register in agents/index.tsx (agentFactories) and in AGENT_TEMPLATES in packages/runtime/src/init.ts.
  4. Map ref -> binary in AGENT_EXECUTABLES in packages/runtime/src/doctor.ts, using surface.binary.
  5. Add tests next to the existing ones: index export, agentFactories entry, no module-level construction, and a generated-adapter load test asserting the env passed through workflowControlChildEnvironment.
  6. Touch packages/config only for a hard constraint the harness genuinely rejects. Never add per-harness env allow-list entries to the runtime layer.

Review gate: under ~100 lines, no fs beyond readFileSync, no fetch, no argv indexing, no usage arithmetic, no session handling, no hardcoded model names or context sizes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Treat this as an umbrella rather than a single implementation task: the independent work is tracked in #659, #660, #661, #662 and #663. For a specific adapter, first inspect smithers agents capabilities and packages/runtime/src/templates/smithers/agents/claude.tsx, then follow the listed registration, doctor mapping and test locations. Done means the selected child meets the tight-adapter review gate and its stated qualification checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, devtools, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.