JuliaArrays / JuliaArrays/AxisArrays.jl

Stackoverflow error when doing cat on array of AxisArrays

Open
#203 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
207
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Using AxisArrays v0.4.4, I encountered a StackOverflow error when doing concatenation with a relatively large array as below:

stackoverflowarray = [AxisArray(rand(500,4)) for i in 1:1048]
cat(stackoverflowarray...; dims=3)

This is actually not an uncommon size in MCMC methods, since 500x4 is just for 4 chains with 500 samples each, and 1048 would be the number of parameters.

Additionally, while doing the cat operation, I will see my RAM consistently climb for about 6GB until the stackoverflow error is thrown. This is probably not the cause though, but it might be correlated/side-effect, because I still have more RAM available, since I have seen julia climb to 12GB at times, and now it is at a total of 7GB when it throws the error.

Lastly, note that if I just do this:

nostackoverflowarray = [AxisArray(rand(500,4)) for i in 1:1048]
cat(nostackoverflowarray...; dims=3)

There is no stackoverflow errors, it finishes in a second, and the operation consumes maybe 25 MB

StackoverflowError: ERROR: StackOverflowError: Stacktrace: [1] _zip_iterate_all(::NTuple{1048,Tuple{Int64,Int64}}, ::NTuple{1048,Tuple{}}) at ./iterators.jl:341 [2] iterate(::Base.Iterators.Zip{NTuple{1048,Tuple{Int64,Int64}}}) at ./iterators.jl:334 [3] sizes(::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:12 [4] matchingdims(::NTuple{1048,AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}}) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:13 [5] _cat(::Int64, ::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:26 [6] cat(::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N; dims::Int64) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:16 [7] top-level scope at REPL[4]:1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the 1,048-element AxisArray example and inspect src/combine.jl, especially sizes, matchingdims, and _cat at the stack-trace locations. Compare it with the plain-array concatenation case; done means cat completes without StackOverflowError and preserves the expected concatenated result.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.