georust / georust/polyline

noisy benchmarks

Open
#41 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
33
Forks
12
PR merge metrics
No merged PRs in 30d

Description

I regularly see 10% swings in performance. Typically if I run the benches 3 times, I'll see a spread of at least 10%.

Is anyone else seeing this? I haven't yet figured out why, but it renders the benchmarks nearly useless if you can't safely measure a 10% change in behavior.

P.S. I noticed this while trying to evaluate #37

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no benchmark files, commands, or tests. Start by reproducing the reported three-run spread and compare the benchmark results while investigating why measurements vary; done means identifying the source of the noise and establishing reliable performance measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.