Validate tool inputs against the tool schema in agent-tools
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start with ToolDefinition.args_model in libs/agent-tools/agent_tools/registry.py and the build_sdk_server adapter that dispatches handlers. Review how agent-tools, hackbot-runtime, and lando-client tests cover malformed tool arguments and existing ToolError rendering. Done means schema type failures become structured feedback without changing semantic checks or omitted-argument behavior.
Written by the indexing model from the issue text.
Description
Tool arguments coming from the model are never checked against the tool's own schema, so malformed input reaches handlers and fails in ways the agent cannot act on.
ToolDefinition.args_model (in libs/agent-tools/agent_tools/registry.py) already builds a pydantic model from each tool's signature and caches it, and input_schema is derived from it and handed to the SDK. But no adapter validates with it — build_sdk_server calls the handler directly:
@sdk_tool(mcp_name, defn.description, defn.input_schema)
async def run(args):
try:
result = await defn.handler(ctx, **args)
except ToolError as e:
...
The schema therefore only guides the model; nothing enforces it.
Why it matters
Two concrete failure modes, using try_server.push's tests: dict[str, list[str]]:
- A bare string (
{"xpcshell": "dom/base/test"}) is iterated character by character, silently producing['', 'a', 'b', 'd', 'e', 'm', 'o', 's', 't']as "paths". Handlers currently need hand-rolledisinstanceguards to avoid this. - A wrong element type (
{"xpcshell": [42]}) gets past a container check and raisesAttributeErrorinside the handler.run()only catchesToolError, so it escapes as an unhandled exception rather than becoming feedback the agent could correct.
Pydantic already rejects both, in default (lax) mode, with no configuration:
bare string -> REJECTED list_type: Input should be a valid list
int in list -> REJECTED string_type: Input should be a valid string
tuple -> OK (normalised to list)
Related symptoms of the same shape elsewhere: #6517 (an LLM returned a list where a string was expected and it crashed on .rfind) and possibly #6493 (unhelpful errors when the model malforms tool arguments).
Proposal
Validate with the existing args_model before dispatching, and turn failures into structured tool errors. Put the helper on ToolDefinition (e.g. defn.validate(args)) rather than inline in the claude-sdk adapter, so a future non-claude-sdk adapter reuses it — registry.py is already the framework-neutral home for args_model.
Handlers then drop their hand-rolled type guards and keep only the semantic checks a type cannot express.
Decisions to make
- Error rendering. Pydantic's text is precise but jargon-y and includes a docs URL (
Input should be a valid list [type=list_type, input_value='dom/base/test', input_type=str]). It should be rendered down to something liketests.xpcshell: Input should be a valid list. Pydantic should own type errors; hand-writtenToolErrors keep owning semantic ones, which can teach the agent what to do instead. - Lax, not strict. Models routinely emit
"123"for an int; lax coerces, strict rejects. Lax also accepts tuple→list, which is wanted. - Validate-only, or pass
model_dump()? Dumping gives handlers coerced values, but also materialises defaults, so a handler can no longer distinguish "omitted" from "explicitlyNone". Validate-then-pass-original is the safer first step. extra.create_modelcurrently ignores unknown arguments, so a hallucinated parameter is silently dropped.extra="forbid"would surface it as feedback, and is the change most likely to break an existing caller.
Scope
This affects every tool across agent-tools and hackbot-runtime, so inputs that currently pass by luck would start failing. Worth its own PR, with the agent-tools, hackbot-runtime and lando-client suites run.
Came up in review of #6531 (https://github.com/mozilla/bugbug/pull/6531#discussion_r3763095103), where a handler needed a manual isinstance check for exactly this reason.
- Dominant language
- Python
- Stars
- 570
- Forks
- 351
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 65
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mozilla/bugbug
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
good-first-bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
hackbot
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
hackbot
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
hackbot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
zostera/django-bootstrap4#894 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
use-agent-os/agent-os#3276 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
zephyrproject-rtos/zephyr#119726 ·
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zilliztech/memsearch#759 ·