microsoft / microsoft/amplifier

Transcript diagnosis is blind to duplicate tool_results: overwrite and masked-intervening-assistant allow corruption to become permanent

Open
#323 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.1k
Forks
261
Avg merge
3h 28m
Merged PRs (30d)
13

Description

Summary

diagnosis.py cannot detect duplicate or out-of-order tool_results because:

  1. build_tool_index uses a dict keyed by tool_call_id — a second result for an already-seen ID silently overwrites the first. The failure mode "duplicate result for one tool_call_id" is not representable; FM1 and FM2 never flag it.

  2. _has_intervening_disruption only treats a tool-less assistant as a disruption — an assistant message that itself issued tool_calls masks the ordering violation, so results that land after such an assistant pass the ordering check even if they belong to an earlier turn.

Consequence: When duplicate/out-of-order tool_results exist (see Issue #322 for how), diagnose() returns "healthy" on every turn while the provider rejects the transcript with 400 unexpected tool_use_id. The corruption becomes permanently undetectable and unrecoverable without manual transcript surgery.

Evidence

Affected session: 8cbc9006-9e2a-4be6-b1a4-151ed5c3575d, transcript lines 117–124

The Corruption
Line 116: assistant [tool_use todo, tool_use delegate]
Line 117–118: tool_result (synthetic placeholder)
Line 119–120: user message  
Line 121: assistant [tool_use bash]  ← THIS masks the ordering violation
Line 122: tool_result (bash result)
Line 123–124: tool_result (REAL todo/delegate results, should be at 117–118)
Why diagnose() Returns "Healthy"

build_tool_index() lines 98–102:

# Builds index of tool_results keyed by tool_call_id
tool_index = {}
for msg in messages:
    if msg.get("role") == "tool":
        tool_index[msg["tool_call_id"]] = msg  # Overwrites if duplicate

When it encounters line 117–118 (the synthetic placeholders), they are added to tool_index. When it encounters line 123–124 (the real results), they overwrite the entries. tool_index ends up with only one entry per ID — the duplicates are invisible.

_has_intervening_disruption() lines 115–125:

def _has_intervening_disruption(...):
    for between_msg in between:
        # Only an assistant WITHOUT tool_calls counts as a disruption
        if between_msg.get("role") == "assistant" and "tool_calls" not in between_msg:
            return True

When checking if results at line 123–124 (which correspond to line 116's tool_calls) are in the right place, it looks at everything between them. Line 121 is an assistant with tool_calls (the bash call), so it returns False — no disruption detected, even though line 121 is a second turn that should be a disruption.

Result

Every call to diagnose() classifies the transcript as healthy:

{
  "status": "healthy",
  "failure_modes": [],
  "orphaned_tool_ids": [],
  "misplaced_tool_ids": [],
  "incomplete_turns": []
}

Meanwhile, every resume attempt fails at the provider:

Anthropic error 400:
"messages.104.content.1: unexpected `tool_use_id` found in `tool_result` blocks: 
toolu_01WshkjDmbu8GReoMdNQiSvQ. Each `tool_result` block must have a corresponding 
`tool_use` block in the previous message."

The session is unresumable and remains undetectable as corrupt.

Root Cause: Three Blind Spots

Blind Spot A — Dict index overwrites duplicates
Line 99: tool_index[msg["tool_call_id"]] = msg silently overwrites. If two results for the same ID exist, the second wins and the first is lost from the index. The check if msg["tool_call_id"] in tool_index: would detect it; a list index (allowing multiple results per ID) would represent it.

Blind Spot B — Intervening-disruption check too narrow
Line 118: if between_msg.get("role") == "assistant" and "tool_calls" not in between_msg: only flags assistants that are NOT making tool calls. An assistant that issues its own tool_calls (like line 121's bash call) is transparent to the check, allowing results from an earlier turn to land after it without raising a flag.

Blind Spot C — No cross-turn-boundary check
There is no validation that ensures "if a tool_result belongs to a tool_call from turn N, it must appear between turn N's assistant and the next user message." Results landing after a later assistant message (which starts a new turn) pass all checks because the checks are local/pairwise, not global.

Proposed Fix Direction

  1. Detect duplicate tool_call_ids: In build_tool_index(), track ALL results in a list (not a dict), or check if tool_call_id in tool_index: and raise a new failure mode "duplicate_tool_result" before overwriting.

  2. Broaden intervention detection: In _has_intervening_disruption(), any assistant message between a tool_use and its result is a disruption, not just assistants without tool_calls:

    if between_msg.get("role") == "assistant":  # ANY assistant, tool_calls or not
        return True
    
  3. Repair by removing late/duplicate results: When FM detects duplicate or misplaced results, repair_transcript() should keep the result adjacent to the assistant that issued the tool_use, and drop any duplicate or late-arriving copies (as we did manually in #8cbc9006).

Cross-Reference

Related: GitHub microsoft/amplifier Issue #322 — the app-cli defects that allowed concurrent executions to append results out of order in the first place.


Note: Issues are disabled in microsoft/amplifier-foundation, so this issue cannot be filed directly. Please route to the amplifier-foundation maintainers or open in microsoft/amplifier with diagnosis.py scope.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in diagnosis.py with build_tool_index and _has_intervening_disruption, then trace how diagnose() and repair_transcript() consume their results. Reproduce the affected transcript from session 8cbc9006-9e2a-4be6-b1a4-151ed5c3575d, lines 117–124. Done means duplicate and late tool_results are reported instead of healthy, and repair keeps the valid result adjacent to its tool_use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.