appandflow / appandflow/stim

Benchmark iOS isolation needs native toolchain and recording compatibility checks

Aberta
#469 6 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
TypeScript
Estrelas
41
Forks
1
Merge médio
1h 25min
PRs com merge (30d)
397

Descrição

## Problem

The benchmark's macOS sandbox-exec isolation passes its filesystem-access probes but is incompatible with iOS build steps that apply a second sandbox. A clean SDK 58 ExpoModulesJSI XCFramework build invokes SwiftPM through nested xcodebuild; package resolution fails with `sandbox-exec: sandbox_apply: Operation not permitted`.

The iOS runner preflight should establish that ordinary native build and recording operations work under the actual isolation profile before accepting timed runs. Failed attempts and private benchmark transcripts are intentionally not included here.

## Evidence

- Current origin/main includes the runner profile in `scripts/agent-benchmark/runner-isolation.mjs`.
- A harmless reproduction under a generated benchmark profile is `/usr/bin/sandbox-exec -f /usr/bin/sandbox-exec -p '(version 1)(allow default)' /usr/bin/true`: exit 71 with the refusal above.
- The SDK 58 installed ExpoModulesJSI `apple/scripts/build-xcframework.sh` calls xcodebuild against a Swift package after clearing stale per-checkout build state. That nested invocation reports the same refusal.
- Native launch artifact preparation alone does not exercise this path if an existing artifact is reused.
- Recording smoke coverage is also needed: agent-device 0.20.10 can report `simctl recordVideo did not expose a complete process identity` in this environment. Its cause needs a separate focused reproduction; it is not yet proven to be the same sandbox limitation.

## Cause

The outer macOS sandbox prevents a child from applying another sandbox. The current probes cover direct/child file access, symlinks, and writes, not native toolchain compatibility. The recording failure's cause is not established.

## Fix idea

Add real untimed iOS toolchain and recording smoke checks under the exact runner policy, and choose a compatible isolation/provisioning design. Preserve forbidden-path denial, per-run device/session ownership, version pins, and identical arm setup. Do not solve this by removing isolation, silently trusting the agent, or relabeling failed timing results as valid. Do not change the frozen campaign until the replacement is reviewed and validated.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.