MegEngine / MegEngine/InferLLM

Thread wakening may be bottom neck for large core systems

Open
#61 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
752
Forks
94
PR merge metrics
No merged PRs in 30d

Description

Result: boost from `3.6~8token/s` to `4.2~4.3token/s` on SG2042.

Analyze: I added tracy to trace execution in details. I observed that worker wakening is problematic. **There are some workers picked up task after the completion of other workers.**

That means the execution time is as twice as the expected sometimes. I guess that it is caused by `thread::yield()`, which switched out the busy waiting loop.

I am thinking about to make a higher/busier performance poll, while being able to switch to an idle state when waiting for user input.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Tracy traces described in the issue and investigate the worker-wakening path around thread::yield(). Reproduce the delayed task pickup on SG2042, then measure whether a busier polling strategy improves throughput without preventing the workers from becoming idle while waiting for user input.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.