juspay / juspay/yama

3.0.4: malformed finalisation burns to the time limit; failure is then misreported as "no tools available"

Open
#83 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
4
Forks
10
Avg merge
11h 34m
Merged PRs (30d)
15

Description

**Version:** `@juspay/yama@3.0.4` · **Provider:** anthropic (`claude-sonnet-5`) via proxy · **Mode:** `pr`, Bitbucket Server · **Config:** `loop.maxSteps: 140`, `ai.timeout: "10m"`

Two related problems, observed on two runs of the same repo, same agent, same config, minutes apart. One run reviewed normally; the other burned ~2.5M tokens over 474s and exited 1 with no verdict.

---

## 1. A malformed finalisation runs to the time limit instead of being retried

On the failing run the model emitted its finalisation **as raw text containing tool-call markup**, rather than as a structured tool call. It survives verbatim in the `summary` field of the run result:

```
...this is a process block, not a code-quality verdict. []
```

Note the `` and `` — that is a tool call the model wrote as prose. Yama never received a verdict, kept looping, and terminated on `"stopReason": "time-limit"`.

Grepping the two build logs for ` no tools were available in this session to fetch the pull request, its diff, or the repository contents, and no code-review or comment-posting tools were exposed

That is not what happened. From the same run's own startup output:

```
🔌 Registering 2 MCP server(s) [mode=pr, role=review]...
✅ bitbucket registered (21 tools)
✅ code-review-graph registered (30 tools)
📊 MCP diagnostics:
Servers: 2 Tools: 57 Connected: 2
✅ MCP servers configured
```

57 tools across 2 connected servers. The "no tools available" claim is the model narrating its own confusion, and it gets surfaced to the user as though it were a diagnostic fact — which sends anyone debugging this straight at MCP configuration, where there is nothing wrong.

Separately, **`statistics.toolCallsMade` reads `0` on both runs** — including the passing one, which reviewed 2 files and posted 4 inline comments. So the counter cannot be used to distinguish "did no work" from "worked fine", which is exactly what it looks like it is for.

**Expected:** don't promote model-authored prose to a diagnostic. When a run ends with `filesReviewed: 0` and a `time-limit` stop reason, say that. And either fix `toolCallsMade` or drop it.

---

## Reproduction

Not deterministic. Both runs were the same version, agent, provider, model and config; the only difference was the diff under review:

- **failing** — 6 files, ~358 insertions, dominated by a Jenkinsfile containing shell with heavily escaped JSON (`\"state\":\"$1\"`) and nested quoting
- **passing** — 2 plain TypeScript files

Offered as a hypothesis only, on a single sample: escape-dense context may make the model more likely to emit tool calls as text. If that is the mechanism, it would be worth a regression fixture containing nested-quote-heavy content.

Happy to supply the full run reports (`.yama/reports/*.md`) and session ids privately if useful.

Contributor guide

Open the contributing guide

Research direction

Start with the supplied .yama/reports/*.md run reports and compare the failing and passing sessions, especially finalisation output, stopReason, filesReviewed, and statistics.toolCallsMade. Trace the finalisation, timeout, diagnostic-summary, and statistics entry points, then add a regression fixture for malformed tool-call markup and verify that the run reports a factual failure without exhausting the time limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ci-cd, devtools, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.