JuliaCI / JuliaCI/BenchmarkTools.jl

Unrealistic values of same tests after repeated executions

Open
#306 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Julia
Stars
684
Forks
107
PR merge metrics
No merged PRs in 30d

Description

The following simple tests behave differently in a second execution.

using BenchmarkTools
using LinearAlgebra
a = rand(300,300); u = triu(rand(300,300));
@btime u*a;      # multiplication without exploiting upper triangular shape
@btime UpperTriangular(u)*a;     # multiplication with exploitation of upper triangular shape
@btime lmul!(UpperTriangular(u),a);  # in-place multiplication
@btime u*a;      # multiplication without exploiting upper triangular shape
@btime UpperTriangular(u)*a;     # multiplication with exploitation of upper triangular shape
@btime lmul!(UpperTriangular(u),a);  # in-place multiplication

These are the results folowing a new start of Julia (other executions produce different figures):

julia> @btime u*a;      # multiplication without exploiting upper triangular shape
  281.900 μs (2 allocations: 703.17 KiB)

julia> @btime UpperTriangular(u)*a;     # multiplication with exploitation of upper triangular shape
  167.200 μs (3 allocations: 703.19 KiB)

julia> @btime lmul!(UpperTriangular(u),view(a,:,:));  # in-place multiplication
  845.800 μs (5 allocations: 176 bytes)

julia> @btime u*a;      # multiplication without exploiting upper triangular shape
  3.432 ms (2 allocations: 703.17 KiB)

julia> @btime UpperTriangular(u)*a;     # multiplication with exploitation of upper triangular shape
  3.011 ms (3 allocations: 703.19 KiB)

julia> @btime lmul!(UpperTriangular(u),view(a,:,:));  # in-place multiplication
  3.045 ms (5 allocations: 176 bytes)

Some values show a more than 10-times slow down. Is any explanation for this behaviour?

julia> versioninfo()
Julia Version 1.8.3
Commit 0434deb161 (2022-11-14 20:14 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported benchmark sequence from the issue on Julia 1.8.3 under Windows, including the repeated calls to u*a, UpperTriangular(u)*a, and lmul!. Compare the first and second results, then determine and document the cause of the reported slowdown; done means the differing timings have a confirmed explanation or a narrowly defined fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.