JuliaCI / JuliaCI/BaseBenchmarks.jl
Noisy benchmarks review
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 44
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
Using https://perf.julialang.org/?tab=benchmarks&bv=noisy I asked claude to review the top offenders
cc. @vtjnash
Top run-to-run noise on the Nanosoldier x86_64 runner over the last ~6 months (Nov 2025 → May 2026), from julia-ci-timing. Noise = median |log(v[i]/v[i-1])| between consecutive runs, as %. Filtered to median latency ≥ 50 ns and noise ≥ 5%.
| Group | Benchmark | Latest | Noise | Likely cause | Action |
|---|---|---|---|---|---|
| sparse | transpose/("adjoint", (20000, 20000)) |
89 ns | 365% | Measures lazy Adjoint wrapper construction, not transpose |
#344 |
| sparse | transpose/("adjoint", (20000, 10000)) |
80 ns | 120% | Same | #344 |
| sparse | transpose/("transpose", (20000, 10000)) |
85 ns | 49% | Same | #344 |
| sparse | transpose/("transpose", (20000, 20000)) |
84 ns | 15% | Same | #344 |
| find | findprev/("Vector{Bool}", "50-50") |
2.3 µs | 194% | Bimodal 775/2330 ns; doesn't repro on aarch64; suspected x86 µop-cache alignment flip | File upstream (JuliaLang/julia) |
| sparse | index/("spmat", "row", "logical", 1000) |
4.6 µs | 14% | Sparse indexing path | Investigate |
| collection | deletion/("Set", "String", "filter") |
27 µs | 14% | String hash bucket order non-deterministic across processes | Use samerandstring |
| collection | deletion/("IdDict", "String", "filter!") |
11 µs | 12% | IdDict ordering depends on String pointer addresses |
Use samerandstring or non-String keys |
| collection | deletion/("Set", "String", "filter!") |
4.9 µs | 11% | Same | Use samerandstring |
| collection | set operations/("Set", "Int", "union", "Set", "Set") |
11 µs | 13% | Set bucket order sensitive to layout | Audit setup determinism |
| collection | set operations/("Set", "Int", "union", "BitSet", "BitSet") |
11 µs | 12% | Same | Audit setup determinism |
| collection | set operations/("Set", "Int", "union", "BitSet") |
11 µs | 12% | Same | Audit setup determinism |
| collection | set operations/("Set", "Int", "union", "Vector") |
11 µs | 12% | Same | Audit setup determinism |
| collection | set operations/("Set", "Int", "union", "Vector", "Vector") |
11 µs | 10% | Same | Audit setup determinism |
| collection | set operations/("Set", "Int", "union", "Set") |
11 µs | 9% | Same | Audit setup determinism |
| union | array/("skipmissing", "perf_sumskipmissing", "Union{Nothing, Int64}", 0) |
4.6 µs | 12% | Union-typed getindex codegen |
Investigate |
| array | index/("sumcolon", SubArray{Float32,2,…ReshapedArray…}) |
6.4 µs | 11% | Likely codegen sensitivity for nested SubArray/ReshapedArray |
Investigate |
| sort | length = 10/mixed eltype with by order |
340 ns | 11% | Tiny mixed-type sort; likely codegen | Probably yellow-flag only |
| array | index/("sumcartesian", ReinterpretArray{Int32,3,Float64,…}) |
815 ns | 10% | Reinterpret indexing codegen | Investigate |
| array | index/("sumcartesian_view", ReinterpretArray{Int32,3,Float64,…}) |
818 ns | 10% | Same | Investigate |
| linalg | arithmetic/("*", Bidiagonal, Vector, 256) |
430 ns | 9% | Sub-µs, near measurement floor | Yellow-flag only |
| linalg | arithmetic/("+", Diagonal, Diagonal, 256) |
395 ns | 9% | Same | Yellow-flag only |
| find | findall/("Vector{Bool}", "90-10") |
860 ns | 9% | Same family as findprev Vector{Bool} |
File upstream |
| find | findall/("Vector{Bool}", "10-90") |
439 ns | 9% | Same | File upstream |
| union | array/("perf_binaryop", "*", "Int8", "(true, true)") |
18 µs | 8% | Union-typed codegen | Investigate |
| sparse | index/("spmat", "row", "array", 1000) |
12 µs | 8% | Sparse indexing | Investigate |
| linalg | blas/gemv! |
240 µs | 7% | Likely runner-side (cache/affinity), not BaseBenchmarks | Runner-side |
| sparse | matmul/("At_mul_Bt!", "dense 40x4000, sparse 40x40 → dense 4000x40") |
3.2 ms | 7% | Same | Runner-side |
Suggested order of attack
- Merge #344 — removes the 4 worst entries.
- Audit
setupblocks inCollectionBenchmarks.jl(and anyString-keyedDict/Set/IdDict) to route all randomness throughRandUtils.samerand/samesprand/samerandstring. This addresses ~7 entries. - File a Julia issue for
findprev/findallonVector{Bool}documenting the bimodality with the dashboard link. - Investigate the
union/SubArray/ReinterpretArraycodegen-sensitive entries individually.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the noisy benchmark entries and the setup blocks in CollectionBenchmarks.jl, then review the RandUtils.samerand, samesprand, and samerandstring paths referenced in the issue. Compare results on the perf.julialang.org dashboard; completion requires a documented decision or follow-up for each investigation, including deterministic setup changes where appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100