noisy benchmarks
- Dominant language
- Rust
- Stars
- 33
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
I regularly see 10% swings in performance. Typically if I run the benches 3 times, I'll see a spread of at least 10%.
Is anyone else seeing this? I haven't yet figured out why, but it renders the benchmarks nearly useless if you can't safely measure a 10% change in behavior.
P.S. I noticed this while trying to evaluate #37
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no benchmark files, commands, or tests. Start by reproducing the reported three-run spread and compare the benchmark results while investigating why measurements vary; done means identifying the source of the noise and establishing reliable performance measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100