monad-developers / monad-developers/ultrafuzz

Config validation must prove the binding it reports is the one execution consumes

Open
#694 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
85
Forks
18
Avg merge
11h 10m
Merged PRs (30d)
194

Description

Problem

A configuration surface that parses, validates and reports a binding is indistinguishable at validate time from one that actually drives execution. Both print a resolved binding, both pass their own tests, and both make validate / doctor green. The difference only shows up at run time, when the reported binding turns out to be inert and the run uses something else entirely.

This is the failure mode that got a provider/harness/model binding schema rejected: it added parsing, validation and reporting for fields that no agent adapter ever read, so the surface was complete, tested, documented — and load-bearing for nothing.

The same trap is open for any future field. Nothing in the repo currently requires a validated config field to be reachable from execution.

Proposal

Make reachability a testable property of the config surface rather than a review-time judgement call:

  1. Every key ultrafuzz validates or reports must be consumed on an execution path — for agent bindings, that means it is read by a generated adapter and turned into a constructor option for the harness class that actually runs. The set of agent-table keys is deliberately closed (auth, api_key_env, config_dir); adding a fourth requires showing the consumer in the same change.
  2. Add a test that walks the keys the config loader accepts for agent tables and asserts each one appears in at least one generated adapter template's read path. A key with no consumer fails the test.
  3. Have validate / doctor report the binding by asking the execution layer what it resolved, not by re-deriving it from the config file. A reporter that re-derives can disagree with reality; one that asks cannot.
  4. Add the reachability question to the adapter review gate: for each new field, name the constructor option it becomes and the harness that receives it.

Acceptance criteria

  • A config key accepted by the loader but read by no execution path fails a test with a message naming the key.
  • validate and doctor derive the reported agent binding from the same code path that constructs the agent.
  • The adapter review checklist requires naming the constructor option each new field maps to.

Non-goals

This is not a new schema, a capability contract, or a binding validator. It is a guard on the config surface that already exists.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the config loader's accepted agent-table keys into the generated adapter templates and the execution path used by validate, doctor, and the harness constructor. Identify where adapter review requirements are documented. Done means unconsumed keys fail with their names, and validate/doctor report the binding resolved by the execution layer.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.