dimforge / dimforge/rapier

PhysicsPipeline::step hangs in join_deferred_bvh_optimize with parallel enabled

Open
#1,008 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
5.7k
Forks
387
Avg merge
4d 23h
Merged PRs (30d)
6

Description

I'm seeing a physics freeze with Rapier's `parallel` feature. I first hit it in a Godot game on an iPhone, then reproduced it on macOS with the standalone Rust program below. The Rust program does not use Godot or any game code.

The reproducer uses rapier3d 0.35.1 with `parallel` and `enhanced-determinism`, rayon 1.12.0 and rayon-core 1.13.0. I tested it on an M1 Pro running macOS 26.1, with rustc 1.98.0.

The original game freezes were on an iPhone 17 Pro Max running iOS 26.6.1, Godot 4.7.1 and godot-rapier-physics v0.35.2. That addon bundles rapier3d 0.35.1 and uses a pool of 2 workers on this phone. It calls the step through `pool.install(|| pipeline.step(...))`, which is also how the reproducer runs it.

When it hangs, the main thread is waiting for `ThreadPool::install` to return. One worker is blocked in `join_deferred_bvh_optimize`, waiting on the channel. The other worker is asleep in Rayon's scheduler. This is the relevant part of the stacks, simplified for readability:

```text
Main thread:
ThreadPool::install
Registry::in_worker_cold
LockLatch::wait_and_reset
_pthread_cond_wait

Worker running the step:
PhysicsPipeline::step_inner
PhysicsPipeline::join_deferred_bvh_optimize
Receiver::recv
Thread::park
semaphore_wait_trap

Other worker:
WorkerThread::wait_until_cold
Sleep::sleep
_pthread_cond_wait
```

Both iPhone reports show this pattern. A fresh `sample` of the frozen Rust program shows it too. The attachment contains selected frames from the reports, with local paths and process identifiers removed.

To reproduce, create the two files below and run:

```bash
cargo run --release -- 2 40
```

The program adds 100 moving cuboids every 300 steps and removes them 200 steps later. It also has a ground collider and 30 fixed boxes. A watchdog reports a hang if no step completes for 8 seconds, then exits with code 2. The second argument is the normal run duration; the watchdog can run past that duration if a step is stuck. `HOLD=1` keeps the process alive after detection so it can be sampled.

Cargo.toml

```toml
[package]
name = "repro_bvh_deadlock"
version = "0.1.0"
edition = "2021"

[dependencies]
rapier3d = { version = "=0.35.1", features = ["parallel", "enhanced-determinism"] }
rayon = "1"

[profile.release]
opt-level = 3
```

src/main.rs

```rust
use rapier3d::prelude::*;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};

fn main() {
let args: Vec = std::env::args().collect();
let threads: usize = args.get(1).and_then(|s| s.parse().ok()).unwrap_or(2);
let max_secs: u64 = args.get(2).and_then(|s| s.parse().ok()).unwrap_or(240);
let burst: usize = args.get(3).and_then(|s| s.parse().ok()).unwrap_or(100);
let period: u64 = args.get(4).and_then(|s| s.parse().ok()).unwrap_or(300);
let pool = rayon::ThreadPoolBuilder::new().num_threads(threads).build().unwrap();
println!("pool threads={} max_secs={} burst={} period={}", threads, max_secs, burst, period);

let progress = Arc::new(AtomicU64::new(0));
{
let progress = progress.clone();
std::thread::spawn(move || {
let mut last = 0u64;
let mut stuck_since = Instant::now();
loop {
std::thread::sleep(Duration::from_millis(200));
let now = progress.load(Ordering::SeqCst);
if now != last { last = now; stuck_since = Instant::now(); }
else if stuck_since.elapsed() > Duration::from_secs(8) {
eprintln!("DEADLOCK: no step completed for 8 s, stuck at step {}", last);
if std::env::var("HOLD").is_ok() { loop { std::thread::sleep(Duration::from_secs(1)); } }
std::process::exit(2);
}
}
});
}

let mut pipeline = PhysicsPipeline::new();
let gravity = Vector::new(0.0, -9.81, 0.0);
let params = IntegrationParameters::default();
let mut islands = IslandManager::new();
let mut bp = DefaultBroadPhase::new();
let mut np = NarrowPhase::new();
let mut bodies = RigidBodySet::new();
let mut colliders = ColliderSet::new();
let mut ij = ImpulseJointSet::new();
let mut mj = MultibodyJointSet::new();
let mut ccd = CCDSolver::new();
let ground = bodies.insert(RigidBodyBuilder::fixed());
colliders.insert_with_parent(ColliderBuilder::cuboid(200.0, 0.5, 200.0), ground, &mut bodies);
// Fixed boxes to give the broad phase some static geometry.
for i in 0..30 {
let h = bodies.insert(RigidBodyBuilder::fixed().translation(Vector::new((i as f32) * 3.0 - 45.0, 2.0, 10.0)));
colliders.insert_with_parent(ColliderBuilder::cuboid(1.0, 2.0, 1.0), h, &mut bodies);
}
let mut seed: u64 = 0x5E17;
let mut rnd = move || { seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407); ((seed >> 33) as f32) / (u32::MAX as f32 / 2.0) };
let mut debris: Vec = Vec::new();
let start = Instant::now();
let mut step: u64 = 0;
loop {
if step % period == 0 {
for _ in 0..burst {
let h = bodies.insert(RigidBodyBuilder::dynamic()
.translation(Vector::new(rnd() * 20.0 - 10.0, 6.0 + rnd() * 4.0, rnd() * 20.0 - 10.0))
.linvel(Vector::new(rnd() * 20.0 - 10.0, rnd() * 10.0, rnd() * 20.0 - 10.0))
.angvel(Vector::new(rnd() * 6.0, rnd() * 6.0, rnd() * 6.0)));
colliders.insert_with_parent(ColliderBuilder::cuboid(0.2 + rnd() * 0.3, 0.2 + rnd() * 0.3, 0.2 + rnd() * 0.3), h, &mut bodies);
debris.push(h);
}
}
if step % period == period * 2 / 3 {
for h in debris.drain(..) {
bodies.remove(h, &mut islands, &mut colliders, &mut ij, &mut mj, true);
}
}
pool.install(|| {
pipeline.step(gravity, ¶ms, &mut islands, &mut bp, &mut np, &mut bodies, &mut colliders, &mut ij, &mut mj, &mut ccd, &(), &());
});
step += 1;
progress.store(step, Ordering::SeqCst);
if step % 20000 == 0 {
println!("{} steps, {:.0} s, {} bodies", step, start.elapsed().as_secs_f32(), bodies.len());
}
if start.elapsed() > Duration::from_secs(max_secs) {
println!("NO DEADLOCK after {} steps in {} s", step, max_secs);
break;
}
}
}
```

These are the results collected on this Mac, including three fresh runs with 2 workers and a longer check with 1 worker:

| Workers | Runs | Result |
| --- | --- | --- |
| 1 | 2 | No hang in runs of 20 and 60 seconds. The latter completed 945,614 steps. |
| 2 | 11 | All hung, after 913 to 53,190 completed steps. |
| 3 | 2 | Both hung, after 24,658 and 27,386 steps. |
| 4 | 2 | Both hung, after 288,713 and 364,916 steps. |
| 6 | 2 | One completed 30 seconds; the longer run hung after 337,862 steps. |
| 8 | 1 | Hung after 975,736 steps. |

These runs show that the hang can occur with more than 2 workers. There aren't enough runs to establish a failure rate for each pool size. With 1 worker, Rapier runs the deferred optimization inline at the join point instead of using the channel. I haven't seen a hang in that configuration during these tests.

Looking at [the scheduling code in 0.35.1](https://docs.rs/rapier3d/0.35.1/src/rapier3d/pipeline/physics_pipeline/solve.rs.html), the deferred optimization is submitted with `rayon::spawn`. The step later waits for it with a blocking `recv()` in [join_deferred_bvh_optimize](https://docs.rs/rapier3d/0.35.1/src/rapier3d/pipeline/physics_pipeline/mod.rs.html). That wait prevents the current worker from running other queued work. I suspect this combination, but the stacks alone don't explain why the other workers remain asleep or prove a lost wakeup. I haven't tested a fix.

The same scheduling and receive code is present in the 0.35.3 sources I checked. The [Godot addon's v0.35.4 lockfile](https://github.com/appsinacup/godot-rapier-physics/blob/v0.35.4/Cargo.lock) still lists rapier3d 0.35.1. I have only run this reproducer against rapier3d 0.35.1.

[piles-a-joindre.txt](https://github.com/user-attachments/files/32070464/piles-a-joindre.txt)
[rapier-bvh-hang-reproducer.zip](https://github.com/user-attachments/files/32070463/rapier-bvh-hang-reproducer.zip)

Contributor guide

Open the contributing guide

Research direction

Start with the supplied reproducer using `cargo run --release -- 2 40`, then inspect `rapier3d/pipeline/physics_pipeline/solve.rs` and `mod.rs`, especially the `rayon::spawn` scheduling and `join_deferred_bvh_optimize` receive. Compare behavior across worker counts and add or run regression coverage if the relevant test entry point is identified. Done means the reproducer no longer hangs while deferred BVH optimization completes correctly with parallel workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
game-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.