facebookexperimental / facebookexperimental/hermit
Port the Buck integration-test matrix to a Cargo-native runner
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The public Cargo workflow does not run Hermit integration tests at the same depth as the internal Buck workflow.
The `tests` and `flaky-tests` workspace members build guest programs, and `cargo test --workspace` runs unit tests plus a small set of Detcore integration tests. It does not run the more than 700 internal Buck scenarios that combine guest binaries with Hermit run modes, schedules, record/replay, chaos settings, and the rr suite.
This issue supersedes the original 2022 proposal to reproduce the internal build with public Buck and Nix. Cargo now builds the ten-crate public workspace successfully and is the simpler contributor-facing path. The integration matrix should be ported to a Cargo-native runner instead of requiring public contributors to reproduce Meta's monorepo build.
## How to observe the gap
On x86_64 Linux with the documented native dependencies installed:
```bash
cargo build --workspace
cargo test --workspace --no-fail-fast
```
The guest binaries compile, but the output does not enumerate the internal matrix represented by the Buck test definitions. A green Cargo run therefore does not provide equivalent coverage.
## Proposed work
- Inventory the internal Buck tests and map each scenario to its guest binary, Hermit mode, arguments, expected output, and hardware requirements.
- Define a Cargo-native integration runner with stable filtering, timeout, logging, and failure-reporting behavior.
- Port a representative vertical slice first: basic run-mode determinism, virtual time, randomness, threading/futex, and record/replay.
- Reuse the existing programs under `tests/` and `flaky-tests/` rather than duplicating fixtures.
- Separate hardware-independent tests from tests requiring PMU retired-conditional-branch counters, CPUID interception, or specific CPU features.
- Run the hardware-independent group in GitHub Actions. Provide a separate documented command and suitable runner strategy for hardware-dependent coverage.
- Track tests that cannot be ported, including unavailable rr inputs or Meta-only infrastructure, in an explicit coverage manifest.
- Provide one local validation command that contributors and agents can run before submitting changes.
## Acceptance criteria
- A documented Cargo command runs the ported end-to-end suite from a clean public checkout.
- GitHub Actions runs the hardware-independent integration group.
- Failures identify the guest, Hermit mode, seed/schedule, command line, and captured logs.
- Hardware-dependent tests use explicit capability detection and are not silently treated as passing.
- The repository contains an inventory mapping every internal Buck scenario to ported, intentionally excluded, or still-pending status.
- The initial port covers at least one scenario for deterministic execution, chaos mode, virtualized time/randomness, threading, and record/replay.
Related environment documentation should be handled alongside the test port so contributors can distinguish regressions from unsupported VM or CPU configurations.
Contributor guide
Assessment
This issue has not been assessed yet.