queryverse / queryverse/DataValues.jl

Reduce invalidations

Open
#84 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
14
Forks
6
Avg merge
1d 7h
Merged PRs (30d)
2

Description

Loading DataValues causes a lot of invalidations, which in turn require a lot of already loaded package code to be recompiled (see at the bottom of this post):

It would be nice to at least fix these methods, which together invalidate about 9,000 methods:

https://github.com/queryverse/DataValues.jl/blob/d568d258d8735e8478b4974dfa0bc5816088c5cb/src/scalar/core.jl#L223
https://github.com/queryverse/DataValues.jl/blob/d568d258d8735e8478b4974dfa0bc5816088c5cb/src/scalar/core.jl#L42 (proposed to be removed in #51)
https://github.com/queryverse/DataValues.jl/blob/d568d258d8735e8478b4974dfa0bc5816088c5cb/src/scalar/core.jl#L224

Here are the invalidations on v1.8:

julia> using SnoopCompileCore

julia> invalidations = @snoopr using DataValues;

julia> using SnoopCompile

julia> trees = invalidation_trees(invalidations)
13-element Vector{SnoopCompile.MethodInvalidations}:
 inserting convert(::Type{Array{S, N}}, X::DataValueArray{T, N}) where {S, T, N} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/primitives.jl:271 invalidated:
   mt_backedges: 1: signature Tuple{typeof(convert), Type{Vector{Any}}, Any} triggered MethodInstance for setindex!(::Vector{Vector{Any}}, ::Any, ::Int64) (0 children)

 inserting |(x::DataValue{Bool}, y::Bool) in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:272 invalidated:
   mt_backedges: 1: signature Tuple{typeof(|), Any, Bool} triggered MethodInstance for Base._base(::Int64, ::Integer, ::Int64, ::Bool) (0 children)

 inserting ^(a::T1, b::DataValue{T2}) where {T1, T2} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/operations.jl:75 invalidated:
   mt_backedges: 1: signature Tuple{typeof(^), String, Any} triggered MethodInstance for OhMyREPL.untokenize_with_ANSI(::IOContext{IOBuffer}, ::Vector{Crayons.Crayon}, ::Vector{Tokenize.Tokens.Token}, ::Any) (0 children)

 inserting &(x::Bool, y::DataValue{Bool}) in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:254 invalidated:
   mt_backedges: 1: signature Tuple{typeof(&), Bool, Any} triggered MethodInstance for div(::Unsigned, ::Int64, ::RoundingMode{:Down}) (0 children)
                 2: signature Tuple{typeof(&), Bool, Any} triggered MethodInstance for div(::Unsigned, ::Int64, ::RoundingMode{:Up}) (0 children)
                 3: signature Tuple{typeof(&), Bool, Any} triggered MethodInstance for Base.var"#string#427"(::Int64, ::Int64, ::typeof(string), ::Unsigned) (0 children)

 inserting mapreduce(f, op::Function, X::T; skipna) where {N, S<:DataValue, T<:AbstractArray{S, N}} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/reduce.jl:109 invalidated:
   backedges: 1: superseding mapreduce(f, op, A::Union{Base.AbstractBroadcasted, AbstractArray}; dims, init) in Base at reducedim.jl:357 with MethodInstance for mapreduce(::Base.ExtremaMap{typeof(identity)}, ::typeof(Base._extrema_rf), ::Vector) (8 children)

 inserting similar(x::AbstractArray, ::Type{DataValue{T}}, dims::Tuple{Vararg{Int64, N}} where N) where T in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/primitives.jl:12 invalidated:
   backedges: 1: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::Type, ::Tuple{Int64}) (3 children)
              2: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::DataType, ::Tuple{Int64}) (3 children)
              3: superseding similar(a::AbstractArray, ::Type{T}, dims::Tuple{Vararg{Int64, N}}) where {T, N} in Base at abstractarray.jl:806 with MethodInstance for similar(::UnitRange{Int64}, ::Type, ::Tuple{Int64}) (6 children)
   1 mt_cache

 inserting convert(::Type{Array}, X::DataValueArray{T, N}) where {T, N} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/primitives.jl:291 invalidated:
   backedges: 1: superseding convert(::Type{T}, a::AbstractArray) where T<:Array in Base at array.jl:617 with MethodInstance for convert(::Type, ::AbstractArray) (15 children)
   29 mt_cache

 inserting convert(::Type{Vector}, X::DataValueVector{T}) where T in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/primitives.jl:283 invalidated:
   mt_backedges: 1: signature Tuple{typeof(convert), Type{Vector}, Any} triggered MethodInstance for Pkg.REPLMode.Command(::Pkg.REPLMode.CommandSpec, ::Dict{Symbol, Any}, ::Any) (1 children)
                 2: signature Tuple{typeof(convert), Type{Vector}, Any} triggered MethodInstance for Pkg.REPLMode.Command(::Nothing, ::Dict{Symbol, Any}, ::Any) (12 children)
   backedges: 1: superseding convert(::Type{T}, a::AbstractArray) where T<:Array in Base at array.jl:617 with MethodInstance for convert(::Type{Vector}, ::AbstractArray) (2 children)

 inserting similar(x::Array, ::Type{DataValue{T}}, dims::Tuple{Vararg{Int64, N}} where N) where T in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/array/primitives.jl:16 invalidated:
   mt_backedges: 1: signature Tuple{typeof(similar), Vector, Any, Tuple{Int64}} triggered MethodInstance for similar(::Vector, ::Tuple{Base.OneTo{Int64}}) (2 children)
   backedges: 1: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pair{DataType, Function}}, ::DataType, ::Tuple{Int64}) (1 children)
              2: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Any}, ::DataType, ::Tuple{Int64}) (1 children)
              3: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector, ::Type, ::Tuple{Int64}) (1 children)
              4: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Array, ::DataType, ::Tuple{Int64}) (2 children)
              5: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Array, ::Type, ::Tuple{Int64}) (2 children)
              6: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Pair{DataType, Function}}, ::Type, ::Tuple{Int64}) (6 children)
              7: superseding similar(a::Array, T::Type, dims::Tuple{Vararg{Int64, N}}) where N in Base at array.jl:378 with MethodInstance for similar(::Vector{Any}, ::Type, ::Tuple{Int64}) (11 children)
   30 mt_cache

 inserting !(x::DataValue{T}) where T<:Number in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:203 invalidated:
   mt_backedges:  1: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#97#98"{typeof(iszero)})(::Any) (0 children)
                  2: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Int64) (0 children)
                  3: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Int64, ::Any) (0 children)
                  4: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::AbstractFloat, ::AbstractFloat) (0 children)
                  5: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.LazilyInitializedFields.lazy_struct(::Expr) (0 children)
                  6: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float64}) (0 children)
                  7: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) (0 children)
                  8: signature Tuple{typeof(!), Any} triggered MethodInstance for showerror(::IOContext{Base.TTY}, ::MethodError) (0 children)
                  9: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Tuple{Base.OneTo{Int64}}, ::Any) (0 children)
                 10: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}, ::Union{Nothing, Pkg.Types.UpgradeLevel, VersionNumber, String, Pkg.Versions.VersionSpec}) (0 children)
                 11: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::typeof(identity), ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::Vector) (0 children)
                 12: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Sort.var"#sort!#8"(::Base.Sort.Algorithm, ::typeof(isless), ::Function, ::Nothing, ::Base.Order.ForwardOrdering, ::typeof(sort!), ::Vector) (0 children)
                 13: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Any) (0 children)
                 14: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Unsigned, ::Int64) (0 children)
                 15: signature Tuple{typeof(!), Any} triggered MethodInstance for ==(::Dict{String, Any}, ::Dict{String, Any}) (0 children)
                 16: signature Tuple{typeof(!), Any} triggered MethodInstance for Pkg.REPLMode._completions(::String, ::Bool, ::Int64, ::Int64) (0 children)
                 17: signature Tuple{typeof(!), Any} triggered MethodInstance for allunique(::AbstractRange) (0 children)
                 18: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Char) (0 children)
                 19: signature Tuple{typeof(!), Any} triggered MethodInstance for ∉(::GlobalRef, ::Any) (0 children)
                 20: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.do_test_throws(::Test.ExecutionResult, ::Any, ::Any) (0 children)
                 21: signature Tuple{typeof(!), Any} triggered MethodInstance for Test.eval_test(::Expr, ::Expr, ::LineNumberNode, ::Bool) (0 children)
                 22: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float16}) (0 children)
                 23: signature Tuple{typeof(!), Any} triggered MethodInstance for !=(::Any, ::Type{Float32}) (0 children)
                 24: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{}) (1 children)
                 25: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair{Symbol, Any}) (1 children)
                 26: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.isdelimited(::IOContext{IOBuffer}, ::Pair) (1 children)
                 27: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Expr, ::Any, ::Expr) (1 children)
                 28: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::LinearAlgebra.var"#249#250") (1 children)
                 29: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.at_disable_library_threading(::Function) (1 children)
                 30: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL.LineEditREPL(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) (1 children)
                 31: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Tuple{Any, Vararg{Any}}) (2 children)
                 32: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::Any, ::Any, ::Any, ::Any, ::Expr) (2 children)
                 33: signature Tuple{typeof(!), Any} triggered MethodInstance for ==(::Vector{Int64}, ::Array) (2 children)
                 34: signature Tuple{typeof(!), Any} triggered MethodInstance for REPL._trimdocs(::Markdown.MD, ::Bool) (2 children)
                 35: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.CoreLogging.log_record_id(::Any, ::Any, ::Any, ::Any) (3 children)
                 36: signature Tuple{typeof(!), Any} triggered MethodInstance for Base._show_nonempty(::IOContext{Base.TTY}, ::AbstractMatrix, ::String, ::Bool, ::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}) (3 children)
                 37: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Pkg.REPLMode.var"#command_is_focused#53"{Bool, Int64})() (3 children)
                 38: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#38#40")(::Core.MethodMatch) (9 children)
                 39: signature Tuple{typeof(!), Any} triggered MethodInstance for (::Base.var"#isword#489")(::Char) (12 children)
                 40: signature Tuple{typeof(!), Any} triggered MethodInstance for Base.Docs.moduledoc(::LineNumberNode, ::Module, ::Any, ::Any, ::Expr) (15 children)

 inserting ==(a::T1, b::DataValue{T2}) where {T1, T2} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:224 invalidated:
   backedges: 1: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Base.UUID, ::Any) (8 children)
              2: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Module, ::Any) (12 children)
              3: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Core.TypeName, ::Any) (14 children)
              4: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Method, ::Any) (14 children)
              5: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Symbol, ::Any) (118 children)

 inserting convert(::Type{Any}, ::DataValue{Union{}}) in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:42 invalidated:
   backedges: 1: superseding convert(::Type{Any}, x) in Base at Base.jl:60 with MethodInstance for convert(::Type{Any}, ::Any) (1934 children)
   1 mt_cache

 inserting ==(a::DataValue{T1}, b::T2) where {T1, T2} in DataValues at /home/sethaxen/.julia/packages/DataValues/N7oeL/src/scalar/core.jl:223 invalidated:
   backedges: 1: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Any, ::FileWatching._FDWatcher) (5 children)
              2: superseding ==(x, y) in Base at Base.jl:116 with MethodInstance for ==(::Any, ::Task) (6616 children)
   10 mt_cache

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the methods linked in src/scalar/core.jl at lines 223, 42, and 224, noting that line 42 is proposed for removal in issue #51. Reproduce the report with Julia's SnoopCompileCore using @snoopr using DataValues, then inspect the invalidation trees. Done means these methods no longer cause the reported large invalidation set, while the package behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.