Characterize model-specific malformed and truncated tool-call output
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Outcome
Identify the component that produces malformed or truncated tool-call output in a managed Hermes run, then route the confirmed defect to its owning implementation issue.
This is an investigation task. The current evidence does not establish a NemoClaw, Hermes, model, provider, or transport defect.
## Reported behavior
One managed Hermes run using a GLM model returned malformed or truncated tool-call output. The comparable flow succeeded with a Nemotron model.
The available evidence does not yet identify whether the response changed at the model, inference-provider API, transport, token-budget, Hermes parser, or user-output boundary.
Related NemoClaw reports cover different confirmed failure shapes:
- #10215 — raw or malformed tool-call JSON reaches user output across multiple agents and providers.
- #8979 — a model emits a tool call as plain text after a bounded validation retry.
- #2731 — a realistic tool surface produces text instead of a structured tool call.
- #8133 — an iteration-limit summary emits a tool call that is never executed.
These issues provide comparison cases. They do not own this model-specific observation.
## Investigation requirements
- Record the exact NemoClaw, Hermes, OpenShell, model, provider, and API-family versions.
- Preserve the exact user prompt, system-prompt class, tool names, tool schemas, message history size, reasoning settings, output-token budget, and streaming setting.
- Capture the provider response before Hermes parsing and the corresponding Hermes event and user-visible output.
- Redact credentials, authorization headers, private endpoints, user data, and internal host details.
- Classify the first failing boundary as one of:
- provider response contains invalid JSON;
- response stops because the output-token budget is exhausted;
- structured tool-call data is emitted as text;
- transport truncates a valid response;
- Hermes parsing or serialization changes a valid response;
- user-output handling exposes an internal representation.
- Repeat the same request shape with the affected model and at least one comparison model.
- Change one variable at a time when testing token budgets, streaming, tool schemas, or model selection.
## Acceptance criteria
- [ ] A sanitized reproduction bundle records every required version and request-shape field.
- [ ] Evidence identifies the first boundary where the response becomes malformed or truncated.
- [ ] The investigation distinguishes token exhaustion from malformed JSON, text-form tool calls, transport truncation, and parser mutation.
- [ ] The affected model result is compared with the same request shape on a comparison model.
- [ ] Repeated runs state whether the failure is deterministic, intermittent, or not reproduced.
- [ ] Captured evidence contains no credential, authorization header, private endpoint, username, hostname, or user document content.
- [ ] If NemoClaw owns the defect, a focused Bug issue contains the minimal reproduction and links this task.
- [ ] If a dependency owns the defect, this task records the owning upstream report and the NemoClaw version or qualification change required.
- [ ] If the issue is not reproduced, this task records the tested matrix and the missing evidence needed before closure.
- [ ] The final issue relationship identifies whether #10215, #8979, #2731, or #8133 is a duplicate, regression source, or unrelated comparison.
## Non-goals
- Claim that one model family is generally incompatible from one observation.
- Change model defaults before the failing boundary is known.
- Add parser heuristics for an unclassified response.
- Record private deployment terminology or credential-bearing evidence.
Contributor guide
Research direction
Start by collecting the exact NemoClaw, Hermes, OpenShell, model, provider, and API-family versions plus the sanitized request shape. Capture the provider response before Hermes parsing, the Hermes event, and user-visible output, then vary one factor at a time to identify the first failing boundary. Done means the failure is classified, compared with another model, repetition is characterized, and the owning issue or missing evidence is recorded.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100