3.0.4: malformed finalisation burns to the time limit; failure is then misreported as "no tools available"
- Dominant language
- TypeScript
- Stars
- 4
- Forks
- 10
- Avg merge
- 11h 34m
- Merged PRs (30d)
- 15
Description
**Version:** `@juspay/yama@3.0.4` · **Provider:** anthropic (`claude-sonnet-5`) via proxy · **Mode:** `pr`, Bitbucket Server · **Config:** `loop.maxSteps: 140`, `ai.timeout: "10m"`
Two related problems, observed on two runs of the same repo, same agent, same config, minutes apart. One run reviewed normally; the other burned ~2.5M tokens over 474s and exited 1 with no verdict.
---
## 1. A malformed finalisation runs to the time limit instead of being retried
On the failing run the model emitted its finalisation **as raw text containing tool-call markup**, rather than as a structured tool call. It survives verbatim in the `summary` field of the run result:
```
...this is a process block, not a code-quality verdict. []
```
Note the `` and `` — that is a tool call the model wrote as prose. Yama never received a verdict, kept looping, and terminated on `"stopReason": "time-limit"`.
Grepping the two build logs for ` no tools were available in this session to fetch the pull request, its diff, or the repository contents, and no code-review or comment-posting tools were exposed
That is not what happened. From the same run's own startup output:
```
🔌 Registering 2 MCP server(s) [mode=pr, role=review]...
✅ bitbucket registered (21 tools)
✅ code-review-graph registered (30 tools)
📊 MCP diagnostics:
Servers: 2 Tools: 57 Connected: 2
✅ MCP servers configured
```
57 tools across 2 connected servers. The "no tools available" claim is the model narrating its own confusion, and it gets surfaced to the user as though it were a diagnostic fact — which sends anyone debugging this straight at MCP configuration, where there is nothing wrong.
Separately, **`statistics.toolCallsMade` reads `0` on both runs** — including the passing one, which reviewed 2 files and posted 4 inline comments. So the counter cannot be used to distinguish "did no work" from "worked fine", which is exactly what it looks like it is for.
**Expected:** don't promote model-authored prose to a diagnostic. When a run ends with `filesReviewed: 0` and a `time-limit` stop reason, say that. And either fix `toolCallsMade` or drop it.
---
## Reproduction
Not deterministic. Both runs were the same version, agent, provider, model and config; the only difference was the diff under review:
- **failing** — 6 files, ~358 insertions, dominated by a Jenkinsfile containing shell with heavily escaped JSON (`\"state\":\"$1\"`) and nested quoting
- **passing** — 2 plain TypeScript files
Offered as a hypothesis only, on a single sample: escape-dense context may make the model more likely to emit tool calls as text. If that is the mechanism, it would be worth a regression fixture containing nested-quote-heavy content.
Happy to supply the full run reports (`.yama/reports/*.md`) and session ids privately if useful.
Contributor guide
Research direction
Start with the supplied .yama/reports/*.md run reports and compare the failing and passing sessions, especially finalisation output, stopReason, filesReviewed, and statistics.toolCallsMade. Trace the finalisation, timeout, diagnostic-summary, and statistics entry points, then add a regression fixture for malformed tool-call markup and verify that the run reports a factual failure without exhausting the time limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100