NVIDIA / NVIDIA/cuvs

[BUG] Rust: default parallel cargo test SIGSEGVs on GPU tests; passes with --test-threads=1

Open
#2,233 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug

Running the Rust crate's test suite with cargo's default parallel test threads SIGSEGVs intermittently on GPU tests. Observed on unmodified main (78135be4): ivf_flat tests crash under the default runner; the same tests pass reliably with --test-threads=1. We reproduced the identical pattern while developing the IVF-SQ (#2229) and refine (#2230) bindings — single-threaded always green, default runner intermittently fatal.

Steps/Code to reproduce bug

cd rust
cargo test -p cuvs              # intermittent SIGSEGV (ivf_flat and others)
cargo test -p cuvs -- --test-threads=1   # passes

Expected behavior

Either the tests are safe to run concurrently, or the harness enforces serialization so contributors don't chase phantom crashes.

Recommended fix

Near-term: enforce serial execution at the harness level rather than relying on contributor knowledge — either a .cargo/config.toml in rust/ setting [env] RUST_TEST_THREADS = "1", or the serial_test crate's #[serial] on GPU-touching tests; plus one line in the rust README. Root-cause follow-up: tests each create their own Resources, so the crash suggests shared global state in the C layer (stream pool, memory resource, or RMM current-device resource) isn't safe under concurrent Resources creation/destruction from multiple host threads — an audit of cuvsResourcesCreate/Destroy thread-safety would pin it down. We're happy to submit the harness-level guard as a small PR; the thread-safety audit likely needs maintainer eyes.

Environment details

Rust crate at main (78135be4), conda libcuvs 26.06, CUDA 13.1, RTX 4000 (8GB), Rust 1.96.0.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/ by running cargo test -p cuvs with and without --test-threads=1, focusing on the ivf_flat GPU tests mentioned in the report. Review the proposed rust/.cargo/config.toml, serial_test option, and the Rust README; done means the default test command no longer intermittently SIGSEGVs and the serialization guidance is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.