appandflow / appandflow/stim

Benchmark iOS isolation needs native toolchain and recording compatibility checks

Aperta
#469 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
TypeScript
Stelle
41
Fork
1
Merge medio
1h 25m
PR unite (30g)
397

Descrizione

## Problem

The benchmark's macOS sandbox-exec isolation passes its filesystem-access probes but is incompatible with iOS build steps that apply a second sandbox. A clean SDK 58 ExpoModulesJSI XCFramework build invokes SwiftPM through nested xcodebuild; package resolution fails with `sandbox-exec: sandbox_apply: Operation not permitted`.

The iOS runner preflight should establish that ordinary native build and recording operations work under the actual isolation profile before accepting timed runs. Failed attempts and private benchmark transcripts are intentionally not included here.

## Evidence

- Current origin/main includes the runner profile in `scripts/agent-benchmark/runner-isolation.mjs`.
- A harmless reproduction under a generated benchmark profile is `/usr/bin/sandbox-exec -f /usr/bin/sandbox-exec -p '(version 1)(allow default)' /usr/bin/true`: exit 71 with the refusal above.
- The SDK 58 installed ExpoModulesJSI `apple/scripts/build-xcframework.sh` calls xcodebuild against a Swift package after clearing stale per-checkout build state. That nested invocation reports the same refusal.
- Native launch artifact preparation alone does not exercise this path if an existing artifact is reused.
- Recording smoke coverage is also needed: agent-device 0.20.10 can report `simctl recordVideo did not expose a complete process identity` in this environment. Its cause needs a separate focused reproduction; it is not yet proven to be the same sandbox limitation.

## Cause

The outer macOS sandbox prevents a child from applying another sandbox. The current probes cover direct/child file access, symlinks, and writes, not native toolchain compatibility. The recording failure's cause is not established.

## Fix idea

Add real untimed iOS toolchain and recording smoke checks under the exact runner policy, and choose a compatible isolation/provisioning design. Preserve forbidden-path denial, per-run device/session ownership, version pins, and identical arm setup. Do not solve this by removing isolation, silently trusting the agent, or relabeling failed timing results as valid. Do not change the frozen campaign until the replacement is reviewed and validated.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.