microsoft / microsoft/amplifier
Transcript diagnosis is blind to duplicate tool_results: overwrite and masked-intervening-assistant allow corruption to become permanent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 261
- Avg merge
- 3h 28m
- Merged PRs (30d)
- 13
Description
Summary
diagnosis.py cannot detect duplicate or out-of-order tool_results because:
-
build_tool_indexuses a dict keyed by tool_call_id — a second result for an already-seen ID silently overwrites the first. The failure mode "duplicate result for one tool_call_id" is not representable; FM1 and FM2 never flag it. -
_has_intervening_disruptiononly treats a tool-less assistant as a disruption — an assistant message that itself issued tool_calls masks the ordering violation, so results that land after such an assistant pass the ordering check even if they belong to an earlier turn.
Consequence: When duplicate/out-of-order tool_results exist (see Issue #322 for how), diagnose() returns "healthy" on every turn while the provider rejects the transcript with 400 unexpected tool_use_id. The corruption becomes permanently undetectable and unrecoverable without manual transcript surgery.
Evidence
Affected session: 8cbc9006-9e2a-4be6-b1a4-151ed5c3575d, transcript lines 117–124
The Corruption
Line 116: assistant [tool_use todo, tool_use delegate]
Line 117–118: tool_result (synthetic placeholder)
Line 119–120: user message
Line 121: assistant [tool_use bash] ← THIS masks the ordering violation
Line 122: tool_result (bash result)
Line 123–124: tool_result (REAL todo/delegate results, should be at 117–118)
Why diagnose() Returns "Healthy"
build_tool_index() lines 98–102:
# Builds index of tool_results keyed by tool_call_id
tool_index = {}
for msg in messages:
if msg.get("role") == "tool":
tool_index[msg["tool_call_id"]] = msg # Overwrites if duplicate
When it encounters line 117–118 (the synthetic placeholders), they are added to tool_index. When it encounters line 123–124 (the real results), they overwrite the entries. tool_index ends up with only one entry per ID — the duplicates are invisible.
_has_intervening_disruption() lines 115–125:
def _has_intervening_disruption(...):
for between_msg in between:
# Only an assistant WITHOUT tool_calls counts as a disruption
if between_msg.get("role") == "assistant" and "tool_calls" not in between_msg:
return True
When checking if results at line 123–124 (which correspond to line 116's tool_calls) are in the right place, it looks at everything between them. Line 121 is an assistant with tool_calls (the bash call), so it returns False — no disruption detected, even though line 121 is a second turn that should be a disruption.
Result
Every call to diagnose() classifies the transcript as healthy:
{
"status": "healthy",
"failure_modes": [],
"orphaned_tool_ids": [],
"misplaced_tool_ids": [],
"incomplete_turns": []
}
Meanwhile, every resume attempt fails at the provider:
Anthropic error 400:
"messages.104.content.1: unexpected `tool_use_id` found in `tool_result` blocks:
toolu_01WshkjDmbu8GReoMdNQiSvQ. Each `tool_result` block must have a corresponding
`tool_use` block in the previous message."
The session is unresumable and remains undetectable as corrupt.
Root Cause: Three Blind Spots
Blind Spot A — Dict index overwrites duplicates
Line 99: tool_index[msg["tool_call_id"]] = msg silently overwrites. If two results for the same ID exist, the second wins and the first is lost from the index. The check if msg["tool_call_id"] in tool_index: would detect it; a list index (allowing multiple results per ID) would represent it.
Blind Spot B — Intervening-disruption check too narrow
Line 118: if between_msg.get("role") == "assistant" and "tool_calls" not in between_msg: only flags assistants that are NOT making tool calls. An assistant that issues its own tool_calls (like line 121's bash call) is transparent to the check, allowing results from an earlier turn to land after it without raising a flag.
Blind Spot C — No cross-turn-boundary check
There is no validation that ensures "if a tool_result belongs to a tool_call from turn N, it must appear between turn N's assistant and the next user message." Results landing after a later assistant message (which starts a new turn) pass all checks because the checks are local/pairwise, not global.
Proposed Fix Direction
-
Detect duplicate tool_call_ids: In
build_tool_index(), track ALL results in a list (not a dict), or checkif tool_call_id in tool_index:and raise a new failure mode"duplicate_tool_result"before overwriting. -
Broaden intervention detection: In
_has_intervening_disruption(), any assistant message between a tool_use and its result is a disruption, not just assistants without tool_calls:if between_msg.get("role") == "assistant": # ANY assistant, tool_calls or not return True -
Repair by removing late/duplicate results: When FM detects duplicate or misplaced results,
repair_transcript()should keep the result adjacent to the assistant that issued the tool_use, and drop any duplicate or late-arriving copies (as we did manually in #8cbc9006).
Cross-Reference
Related: GitHub microsoft/amplifier Issue #322 — the app-cli defects that allowed concurrent executions to append results out of order in the first place.
Note: Issues are disabled in microsoft/amplifier-foundation, so this issue cannot be filed directly. Please route to the amplifier-foundation maintainers or open in microsoft/amplifier with diagnosis.py scope.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in diagnosis.py with build_tool_index and _has_intervening_disruption, then trace how diagnose() and repair_transcript() consume their results. Reproduce the affected transcript from session 8cbc9006-9e2a-4be6-b1a4-151ed5c3575d, lines 117–124. Done means duplicate and late tool_results are reported instead of healthy, and repair keeps the valid result adjacent to its tool_use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100