JuliaArrays / JuliaArrays/StructArrays.jl

Large number of allocations in 0.6.8

Open
#228 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
402
Forks
49
PR merge metrics
No merged PRs in 30d

Description

The eager automatic conversion introduced in #227 (cc @timholy @piever) leads to many allocations when this conversion isn't actually needed.
For example:
StructArrays@0.6.7 - 25 allocations:

julia> using StructArrays, ArraysOfArrays

julia> A = StructArray(x=VectorOfVectors{Int}())

julia> @time for _ in 1:10^6 push!(A, (x=1:10,)) end
  0.307580 seconds (25 allocations: 11.269 MiB, 7.45% gc time)

StructArrays@0.6.8 - 1 million allocations:

julia> using StructArrays, ArraysOfArrays

julia> A = StructArray(x=VectorOfVectors{Int}())

julia> @time for _ in 1:10^6 push!(A, (x=1:10,)) end
  0.809896 seconds (1.00 M allocations: 293.601 MiB, 55.02% gc time)

Of course, these allocations go away if automatic conversion (https://github.com/JuliaArrays/StructArrays.jl/blob/1581d70090de650646277e4eee4e33794e55b9ac/src/utils.jl#L197-L199) is disabled by StructArrays.maybe_convert_elt(::Type{T}, vals::Tuple) where {T} = vals.

This change in 0.6.8 is highly breaking performance-wise. Can the new automatic conversion approach be revised and made less eager somehow?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the allocation benchmark in the issue, comparing StructArrays 0.6.7 and 0.6.8. Inspect the automatic-conversion logic in src/utils.jl at the referenced lines and the behavior of maybe_convert_elt(::Type{T}, vals::Tuple). Done means revising the conversion behavior so the reported push! workload no longer incurs one million allocations while required conversion still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.