Cyphrme / Cyphrme/Cyphr

Workspace test suite has a cross-crate race isolated to cyphr-server's presence in the build

Open
#79 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3
Forks
0
Avg merge
1d 42m
Merged PRs (30d)
11

Description

## Summary

Two tests in `rs/cyphr-storage/tests/crash_recovery.rs` (asserting that the storage engine automatically recovers from a specific crash-window scenario at open) pass reliably every time when run in isolation, or alongside any subset of the workspace that excludes `cyphr-server` — but fail deterministically when `cyphr-server`'s own tests are included in the same `cargo test` invocation. This is a genuine, reproducible cross-crate interaction, not a flake in the traditional sense (it reproduced 2/2 times with nothing else building concurrently, and passed consistently in every narrower configuration tried).

## Root cause (identified, not yet confirmed by direct experiment)

`cyphr-storage`'s own `Cargo.toml` requests only tokio features `[io-util, sync]`. But `cargo tree -i tokio -e features` shows that when `cyphr-server` is built in the same invocation (via its axum/hyper dependency chain), Cargo's feature unification compiles the single shared `tokio` rlib with `net`, `time`, `socket2`, and `rt-multi-thread` added — features `cyphr-storage` never requested itself, but whose compiled artifact its own test binaries then link against.

tokio's `time` feature is documented to change runtime internals (it compiles in an internal timer driver that doesn't exist without it), which is a plausible source of async-timing sensitivity. The specific gate this affects is a `tokio::sync::OnceCell::get_or_try_init`-based lazy initialization guard (added to close a different, unrelated crash-recovery gap) — plausible that a subtle scheduling difference under the richer tokio build exposes a timing edge in that gate that the leaner build doesn't.

## What's confirmed vs. what's still open

Confirmed via bisection:
- Passes: isolated single test file (both single-threaded and default parallelism, multiple runs), all of `cyphr-storage`'s own tests together, `cyphr` + `cyphr-storage` combined, full workspace minus `cyphr-cli` and `cyphr-server`.
- Fails: full workspace, and workspace minus only `cyphr-cli` (i.e., whenever `cyphr-server` is present).
- Ruled out: a stale-build/shared-target-directory compile race (this is a clean logical assertion failure, not a compile error); shared static/global state in the in-memory test doubles involved (reviewed directly — pure per-instance state); tokio's `test-util` feature (not requested anywhere in the workspace); concurrent test-binary execution (cargo runs test binaries strictly sequentially, confirmed from log ordering).

Not yet done: directly forcing tokio's `net`/`time` features on for `cyphr-storage` alone to confirm the reproduction in isolation (which would conclusively pin the mechanism rather than leave it as the leading, well-evidenced hypothesis).

## Why this isn't blocking

The two affected tests pass reliably on their own, and the implementation they exercise has been independently verified correct. This is infrastructure/build-configuration debt, not a defect in the tested code — tracked here rather than gating the correctness work that surfaced it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with rs/cyphr-storage/Cargo.toml, rs/cyphr-storage/tests/crash_recovery.rs, and the reported cargo tree -i tokio -e features output. Force tokio's net/time features for cyphr-storage alone and rerun the affected crash-recovery tests, then compare isolated and workspace runs. Done means the feature-unification hypothesis is confirmed or rejected and the resulting build-configuration issue is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.