🐛 E2E Phase 4: Model ignores loaded skill, brute-forces filesystem search causing hang

Summary

After session restart with skills loaded (Phase 3 -> Phase 4), the model ignores the loaded skill content and instead launches a chain of built-in harness tools (find, list_dir, grep_search) to search the filesystem. Each find command times out after ~30s, and the model retries in different directories indefinitely.

Impact

• E2E test Phase 4 hangs for 3+ minutes (until wall-clock timeout)
• First attempt consistently fails; second attempt (with fresh session) sometimes works
• The 180s idle timeout never fires because each tool response resets the timer

Root Cause Analysis

Timeline of a failing Phase 4 run (from DEBUG trace)

src=SOURCE_USER tgt=TARGET_MODEL state=STATE_DONE td_len=65    # prompt sent
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo files"   # model calls find!
src=SOURCE_MODEL tgt=TARGET_USER state=STATE_DONE td_len=0     # DONE but NO text
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="List root directory"
src=SOURCE_MODEL tgt=TARGET_USER STATE_ERROR ERR="Find command timed out"
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo in Users"
src=SOURCE_MODEL tgt=TARGET_USER STATE_ERROR ERR="Find command timed out"
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo in ricc home"
src=SOURCE_MODEL tgt=TARGET_USER STATE_ERROR ERR="Find command timed out"
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo in Documents"
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo in Desktop"
src=SOURCE_MODEL tgt=TARGET_ENVIRONMENT TD="Find todo in obsidian-pbt26"
# ... continues indefinitely

Why the model ignores the skill

The skill IS loaded (Phase 5 confirms riccardo-todo in agent.skills). However:

1. The harness provides built-in tools (find, list_dir, grep_search) by default
2. The model sees these tools and decides filesystem search is more reliable than skill content
3. Each find command times out on large directories (~/), but the model keeps retrying

Why it hangs instead of timing out

• The timeout: parameter in each_message is an idle timeout (time between messages)
• Each tool call/response generates WebSocket messages, resetting the idle timer
• Total wall-clock time is unbounded

Mitigations Applied (v0.4.1)

1. System instruction guard

system_instruction: "...do NOT use find, list_dir, grep_search or any filesystem tools. " \
                    "The skill already contains the answer."

2. Explicit Phase 4 prompt

"According to the riccardo-todo skill you have loaded, " \
"where is Riccardo's to-do list file stored? " \
"Answer based on the skill instructions only"

3. Wall-clock timeout (Timeout.timeout)

Timeout.timeout(wall_timeout, Timeout::Error, "Wall-clock timeout after #{wall_timeout}s") do
  @agent.ask(text, timeout: wall_timeout) { ... }
end

4. Retry with session reset (up to 3 attempts)

Fresh session on timeout/error, since the model sometimes works on attempt 2.

5. Dynamic TUI status via hooks.on(:ws_message)

Shows real-time activity so the user sees tool loops instead of a blank screen.

Remaining Issues

• Model still frequently ignores skill content on first attempt (~50% of runs)
• Even with "do NOT search filesystem", the model sometimes still calls find
• Wall-clock timeout is a band-aid; root fix would be limiting built-in tool access per agent
• No way to disable specific harness built-in tools from the SDK

Proposed Long-term Fixes

1. SDK config to disable built-in tools: Agent.new(harness_tools: :none) or allowlist
2. Total deadline in collect_response: Native max_duration: parameter instead of wrapping with Timeout.timeout
3. Tool call limit: max_tool_calls: 5 to prevent infinite loops
4. Harness-side skill activation: Model should auto-read skill content without needing tool calls

Commits

• b1c2fb5 Fix e2e Phase 4 prompt + retry + variable fix
• 931a7e2 v0.4.1: Generic event hooks (on/emit)
• 3c57b23 Dynamic TUI status + 60s wall-clock timeout
• 15cf1af Dynamic TUI with state emojis
• a87774a Step counter label

Environment

• macOS, Ruby 3.4.5
• antigravity-ruby-sdk v0.4.1
• Harness: localharness (Antigravity)

Start with the E2E Phase 4 flow, Agent.new, @agent.ask, each_message, and collect_response, then review the listed commits and existing timeout/retry mitigations. A complete fix should prevent unbounded built-in tool loops and enforce a bounded response duration or tool policy without relying only on the current band-aid.