callstack / callstack/agent-device

investigate(ios): confirm historical XCTest tap-failure storm root cause

Open
#1,599 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
4.6k
Forks
299
Avg merge
10h 14m
Merged PRs (30d)
537

Description

## Purpose

Confirm or falsify the remaining historical root-cause hypothesis for the iOS `XCTEST_RECORDED_FAILURE` tap storm seen on hostile/profile-class screens.

The user-facing retry-spiral defect is already fixed:

- #1605 corroborates ambiguous recorded tap failures against a fresh, comparable post-action capture and returns success-with-warning when the tap landed.
- #1634 keeps corroboration on the baseline snapshot backend, covering the private-AX/tree penalty-boundary mismatch seen in the live Bluesky repro.
- After #1587, repeated live attempts could no longer reproduce the natural failure storm.

This issue no longer owns tap-outcome recovery, generic runner restart behavior, or additional retry logic.

## Remaining question

Was the archived failure storm caused by XCTest-channel saturation and abandoned capture work before #1587, or does an independent current defect remain?

## Investigation contract

1. Reproduce against the archived AppControlBench `bsky-24` evidence or a controlled pre-#1587 build using the same profile-screen action sequence.
2. Compare a pre-#1587 run with current main under the same app state and command sequence.
3. Retain runner logs and request diagnostics covering:
- capture duration and penalty state;
- abandoned or coalesced runner work;
- `XCTEST_RECORDED_FAILURE` timing;
- restart/relaunch boundaries;
- whether the UI changed despite the reported failure.
4. Distinguish causal saturation evidence from correlation. A changing timeout/failure set or a non-equivalent app state is not confirmation.

## Completion

- **Confirmed historical cause:** record the matched evidence showing the storm disappears when the #1587 channel-pressure change is applied, then close this issue.
- **Independent current defect:** provide a reproducible current-main sequence and open a narrowly scoped implementation issue for its owning boundary; do not expand this historical investigation into a speculative fix.
- **Not reproducible:** record the bounded attempts and evidence limits, then close as superseded by #1587, #1605, and #1634.

No production change is required unless a distinct current-main defect is reproduced.

Contributor guide

Open the contributing guide

Research direction

Start with the archived AppControlBench bsky-24 evidence or a controlled pre-#1587 build, then compare the same action sequence with current main. Retain runner logs and request diagnostics for capture timing, abandoned work, failure timing, restart boundaries, and UI changes; done means recording confirmed causation, a reproducible current defect, or bounded non-reproduction before closing.

Written by the indexing model from the issue text.

Assessment

Tech stack
ios, typescript
Domain
mobile, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.