MultithreadedExecutor bottlenecking at 1000+ Systems
- Dominant language
- Rust
- Stars
- 48.2k
- Forks
- 4.8k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 171
Description
## Bevy version
0.12.1
## Relevant system information
- CPU: AMD Ryzen Threadripper 3970X 32-Core Processor 3.69 GHz
- RAM: 32GB
## What you did
Hello! I have a use-case that essentially involves separating identical groups of entities. Since Bevy's [subworld support](https://github.com/bevyengine/rfcs/pull/16) is not complete and Bevy does not have shared components (like Unity DOTS), I opted for a solution where I use Rust generics to "duplicate" my systems for every group with a `SpareSet` marker component. So `Marker::<0>, Marker::<1>, ...` components and `SystemA::<0>, SystemA::<1>, ...` systems. The idea was the separate systems/marker components will allow Bevy to properly parallelize logic across groups since there are no cross-group dependencies.
## What went wrong
It seems Bevy is bottlenecked by the number of systems for my use-case. Attempting 6000 systems (2000 groups, 3 systems/group) results in 7% CPU utilization with 12 FPS. A Tracy capture indicates that 80+% of the CPU time is spent in the `multithreaded executor` before sending tasks to my thread pool.
I have created a Github with the capture and code https://github.com/UsaidPro/BevyLotsOfSystems
I was hoping Bevy would distribute the systems across the full thread pool provided by my 32-core CPU. However, instead what happens is 1 core gets consumed by the `multithreaded executor` which _does_ distribute the tasks across all threads (I see 55+ thread pools in Tracy) but only after taking ~60+ms (80+% of compute time). The multithreaded executor has MTPC of 470us, but it is called 17k times compared to 129 Update calls resulting in 83% of time spent in the single thread.
Here is a table of what systems vs FPS. All these used only 7% of my CPU, same bottleneck. I have 3 systems, 1 of them only runs if `run_if()` returned true.
| Groups | Concurrent Systems | Conditional Systems | FPS |
| --- | --- | --- | --- |
| 2000 | 4000 | 2000 | 12 |
| 1000 | 2000 | 1000 | 40 |
| 500 | 1000 | 500 | 60 |
## Additional information
Tracy screenshot:

- [Tracy capture](https://github.com/UsaidPro/BevyLotsOfSystems/blob/master/bevy_capture.tracy)
- [Github with code](https://github.com/UsaidPro/BevyLotsOfSystems). Uses Bevy Rapier3D, which does not seem to be related to this issue.
- [Line in Github where you can set the # of groups you want](https://github.com/UsaidPro/BevyLotsOfSystems/blob/master/RepeatMacro/src/lib.rs#L28)
- This code may be used as a stress-test for Bevy's scheduler handling lots of systems. Can raise a PR if it would be useful.
Contributor guide
Research direction
Start with the linked reproduction and Tracy capture, then inspect RepeatMacro/src/lib.rs at line 28 to understand how the number of groups and systems is configured. Reproduce the reported scaling behavior and trace the MultithreadedExecutor overhead; done means identifying and addressing the scheduler bottleneck for workloads with thousands of systems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- game-dev, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100