bevyengine / bevyengine/bevy

MultithreadedExecutor bottlenecking at 1000+ Systems

Open
#11,378 7 comments 2 reactions 0 assignees View on GitHub
A-ECS C-Bug C-Performance
Dominant language
Rust
Stars
48.2k
Forks
4.8k
Avg merge
3d 16h
Merged PRs (30d)
171

Description

## Bevy version

0.12.1

## Relevant system information

- CPU: AMD Ryzen Threadripper 3970X 32-Core Processor 3.69 GHz
- RAM: 32GB

## What you did

Hello! I have a use-case that essentially involves separating identical groups of entities. Since Bevy's [subworld support](https://github.com/bevyengine/rfcs/pull/16) is not complete and Bevy does not have shared components (like Unity DOTS), I opted for a solution where I use Rust generics to "duplicate" my systems for every group with a `SpareSet` marker component. So `Marker::<0>, Marker::<1>, ...` components and `SystemA::<0>, SystemA::<1>, ...` systems. The idea was the separate systems/marker components will allow Bevy to properly parallelize logic across groups since there are no cross-group dependencies.

## What went wrong

It seems Bevy is bottlenecked by the number of systems for my use-case. Attempting 6000 systems (2000 groups, 3 systems/group) results in 7% CPU utilization with 12 FPS. A Tracy capture indicates that 80+% of the CPU time is spent in the `multithreaded executor` before sending tasks to my thread pool.
I have created a Github with the capture and code https://github.com/UsaidPro/BevyLotsOfSystems

I was hoping Bevy would distribute the systems across the full thread pool provided by my 32-core CPU. However, instead what happens is 1 core gets consumed by the `multithreaded executor` which _does_ distribute the tasks across all threads (I see 55+ thread pools in Tracy) but only after taking ~60+ms (80+% of compute time). The multithreaded executor has MTPC of 470us, but it is called 17k times compared to 129 Update calls resulting in 83% of time spent in the single thread.

Here is a table of what systems vs FPS. All these used only 7% of my CPU, same bottleneck. I have 3 systems, 1 of them only runs if `run_if()` returned true.

| Groups | Concurrent Systems | Conditional Systems | FPS |
| --- | --- | --- | --- |
| 2000 | 4000 | 2000 | 12 |
| 1000 | 2000 | 1000 | 40 |
| 500 | 1000 | 500 | 60 |

## Additional information

Tracy screenshot:
![tracy_screenshot](https://github.com/bevyengine/bevy/assets/7944753/1b50954a-6bca-4f0b-b78c-c856cbbb9b13)

- [Tracy capture](https://github.com/UsaidPro/BevyLotsOfSystems/blob/master/bevy_capture.tracy)
- [Github with code](https://github.com/UsaidPro/BevyLotsOfSystems). Uses Bevy Rapier3D, which does not seem to be related to this issue.
- [Line in Github where you can set the # of groups you want](https://github.com/UsaidPro/BevyLotsOfSystems/blob/master/RepeatMacro/src/lib.rs#L28)
- This code may be used as a stress-test for Bevy's scheduler handling lots of systems. Can raise a PR if it would be useful.

Contributor guide

Open the contributing guide

Research direction

Start with the linked reproduction and Tracy capture, then inspect RepeatMacro/src/lib.rs at line 28 to understand how the number of groups and systems is configured. Reproduce the reported scaling behavior and trace the MultithreadedExecutor overhead; done means identifying and addressing the scheduler bottleneck for workloads with thousands of systems.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
game-dev, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.