voidzero-dev / voidzero-dev/vite-plus

PTY snapshot suite: rare non-repeating case failures under parallel load (pass in isolation)

Open
#2,565 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
5.8k
Forks
261
Avg merge
1d 34m
Merged PRs (30d)
135

Description

## Summary

Running the full `migration*` PTY snapshot suite in parallel occasionally fails a small, **non-repeating** set of cases (~0.4% of case-executions). Every affected case passes deterministically when run in isolation, and the failing names differ between runs — so this looks like load-sensitivity in the PTY runner environment rather than any individual fixture or product bug. Filing per m0g3r's suggestion in #2483 after it showed up there first.

## Observations (macOS arm64, M-series, `just snapshot-test migration`)

Five full-suite runs across two commits, all with a fresh `packages/cli/dist` (freshness guard green):

| run | commit | result | failed cases |
|---|---|---|---|
| A | `31163c5` (PR #2483 branch) | 151/154 | `migration_dynamic_oxc_configs`, `migration_framework_shim_vue`, `migration_standalone_yarn4_idempotent` |
| B | `cc20535d` (main) | 159/161 | `migration_not_supported_vitest3`, `migration_from_tsup_monorepo_success` |
| C | `cc20535d` (main) | 161/161 | — |
| D | `cc20535d` (main) | 161/161 | — |
| E | `31163c5`, earlier same-day | 148/154 | different set again (incl. `migration_husky_or_prepare`, `migration_standalone_bun_install`) |

- **No case name repeats across runs.** Union of failures over five runs: 10+ distinct fixtures, each failing exactly once.
- **Every one of them passes in isolation** (`just snapshot-test `), immediately after the failing run, unchanged tree.
- The failure-heavy runs were the **first suite run after a fresh build / cold caches**; warm re-runs (C, D) were fully green. Cold-state work on first touch (managed runtime provisioning, registry-bridge first hits) overlapping with ~14 parallel PTY cases is my best guess at the mechanism, but I have not isolated it.
- m0g3r additionally verified in #2483 that the flaked fixtures are structurally unremarkable (step counts mid-pack; one of 21 double-`vp migrate` fixtures).

## Why it may matter

Locally it's a shrug; in CI a 0.4% per-case flake across ~650 cases makes a meaningful fraction of runs red for reasons unrelated to the diff under test, and the changing names make it hard for contributors to tell signal from noise (it cost a round of triage on #2483).

## Environment

- macOS arm64 (Darwin 25.2), 14 threads reported by the runner
- `just snapshot-test migration` (both flavors), toolchain per `rust-toolchain.toml`

Happy to re-run with any added instrumentation (timings, per-case retries, runner verbosity) if that helps narrow it — I can reproduce roughly one flaky run per two or three cold full-suite runs.

Contributor guide

Open the contributing guide

Research direction

Start with `just snapshot-test migration` and the PTY snapshot runner, using `rust-toolchain.toml` and the listed migration fixtures as the reproduction surface. Compare cold-cache parallel runs with isolated and warm runs, adding the suggested timing, verbosity, or retry instrumentation. Done means the load-sensitive failures are explained and the migration suite no longer produces unexplained intermittent failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.