appandflow / appandflow/stim

Benchmark iOS isolation needs native toolchain and recording compatibility checks

Offen
#469 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
TypeScript
Sterne
41
Forks
1
Ø Merge
1 Std. 25 Min.
Gemergte PRs (30 T.)
397

Beschreibung

## Problem

The benchmark's macOS sandbox-exec isolation passes its filesystem-access probes but is incompatible with iOS build steps that apply a second sandbox. A clean SDK 58 ExpoModulesJSI XCFramework build invokes SwiftPM through nested xcodebuild; package resolution fails with `sandbox-exec: sandbox_apply: Operation not permitted`.

The iOS runner preflight should establish that ordinary native build and recording operations work under the actual isolation profile before accepting timed runs. Failed attempts and private benchmark transcripts are intentionally not included here.

## Evidence

- Current origin/main includes the runner profile in `scripts/agent-benchmark/runner-isolation.mjs`.
- A harmless reproduction under a generated benchmark profile is `/usr/bin/sandbox-exec -f /usr/bin/sandbox-exec -p '(version 1)(allow default)' /usr/bin/true`: exit 71 with the refusal above.
- The SDK 58 installed ExpoModulesJSI `apple/scripts/build-xcframework.sh` calls xcodebuild against a Swift package after clearing stale per-checkout build state. That nested invocation reports the same refusal.
- Native launch artifact preparation alone does not exercise this path if an existing artifact is reused.
- Recording smoke coverage is also needed: agent-device 0.20.10 can report `simctl recordVideo did not expose a complete process identity` in this environment. Its cause needs a separate focused reproduction; it is not yet proven to be the same sandbox limitation.

## Cause

The outer macOS sandbox prevents a child from applying another sandbox. The current probes cover direct/child file access, symlinks, and writes, not native toolchain compatibility. The recording failure's cause is not established.

## Fix idea

Add real untimed iOS toolchain and recording smoke checks under the exact runner policy, and choose a compatible isolation/provisioning design. Preserve forbidden-path denial, per-run device/session ownership, version pins, and identical arm setup. Do not solve this by removing isolation, silently trusting the agent, or relabeling failed timing results as valid. Do not change the frozen campaign until the replacement is reviewed and validated.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.