sgl-project / sgl-project/SpecForge
High results fluctuation in repeating experiments with same setup
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 347
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 41
Description
I am using the command CUDA_VISIBLE_DEVICES=0,1 python3 SpecForge/benchmarks/bench_eagle3.py --model-path Qwen/Qwen3-8B --speculative-draft-model-path path/to/model --port 30000 --trust-remote-code --mem-fraction-static 0.8 --tp-size 1 --attention-backend fa3 --config-list 1,6,10,32 --benchmark-list mtbench --dtype bfloat16 to test 2 methods, however the first time I tested, baselineA results in TPS 106, baselineB results in TPS 100, the second time, baselineA results in TPS93, baseline B results in TPS 107, the third and later time I tested, both baselines are TPS 93......I don't think the randomness in the experiment is the cause, and the GPU and parameters for each experiment is all the same, does anyone faced the same issue or know about possible reasons?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SpecForge/benchmarks/bench_eagle3.py and reproduce the repeated command using the stated model, GPU settings, and benchmark configuration. Compare the TPS results across repeated runs and trace the benchmark setup or runtime state that differs. Done means identifying a reproducible cause for the fluctuation or documenting the conditions under which it occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100