sgl-project / sgl-project/SpecForge

High results fluctuation in repeating experiments with same setup

Open
#550 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
347
Avg merge
4d 1h
Merged PRs (30d)
41

Description

I am using the command CUDA_VISIBLE_DEVICES=0,1 python3 SpecForge/benchmarks/bench_eagle3.py --model-path Qwen/Qwen3-8B --speculative-draft-model-path path/to/model --port 30000 --trust-remote-code --mem-fraction-static 0.8 --tp-size 1 --attention-backend fa3 --config-list 1,6,10,32 --benchmark-list mtbench --dtype bfloat16 to test 2 methods, however the first time I tested, baselineA results in TPS 106, baselineB results in TPS 100, the second time, baselineA results in TPS93, baseline B results in TPS 107, the third and later time I tested, both baselines are TPS 93......I don't think the randomness in the experiment is the cause, and the GPU and parameters for each experiment is all the same, does anyone faced the same issue or know about possible reasons?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with SpecForge/benchmarks/bench_eagle3.py and reproduce the repeated command using the stated model, GPU settings, and benchmark configuration. Compare the TPS results across repeated runs and trace the benchmark setup or runtime state that differs. Done means identifying a reproducible cause for the fluctuation or documenting the conditions under which it occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.