trixi-framework / trixi-framework/Trixi.jl

Some elixirs fail with `StackOverflowError` on multiple threads

Open
#1,301 0 comments 0 reactions 0 assignees View on GitHub
bug parallelization upstream
Dominant language
Julia
Stars
731
Forks
166
Avg merge
2d 18h
Merged PRs (30d)
25

Description

For example:
```julia
julia> include("examples/p4est_3d_dgsem/elixir_euler_circular_wind_nonconforming.jl")
```
```
████████╗██████╗ ██╗██╗ ██╗██╗
╚══██╔══╝██╔══██╗██║╚██╗██╔╝██║
██║ ██████╔╝██║ ╚███╔╝ ██║
██║ ██╔══██╗██║ ██╔██╗ ██║
██║ ██║ ██║██║██╔╝ ██╗██║
╚═╝ ╚═╝ ╚═╝╚═╝╚═╝ ╚═╝╚═╝

┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ SemidiscretizationHyperbolic │
│ ════════════════════════════ │
│ #spatial dimensions: ………………………… 3 │
│ mesh: ………………………………………………………………… P4estMesh{3, Float64} │
│ equations: …………………………………………………… CompressibleEulerEquations3D │
│ initial condition: ……………………………… initial_condition_circular_wind │
│ boundary conditions: ………………………… 2 │
│ │ inside: ……………………………………………………… typeof(boundary_condition_slip_wall) │
│ │ outside: …………………………………………………… typeof(boundary_condition_slip_wall) │
│ source terms: …………………………………………… source_terms_circular_wind │
│ solver: …………………………………………………………… DG │
│ total #DOFs: ……………………………………………… 54000 │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
[...]

────────────────────────────────────────────────────────────────────────────────────────────────────
Simulation running 'CompressibleEulerEquations3D' with DGSEM(polydeg=4)
────────────────────────────────────────────────────────────────────────────────────────────────────
#timesteps: 0 run time: 7.00000000e-07 s
Δt: 0.00000000e+00 time/DOF/rhs!: NaN s
sim. time: 0.00000000e+00
#DOF: 54000
#elements: 432
└── level 0: 432

Variable: rho rho_v1 rho_v2 rho_v3 rho_e
L2 error: 2.20459321e-16 1.12109309e-15 1.12171013e-15 0.00000000e+00 1.04810638e-05
Linf error: 5.55111512e-16 8.88178420e-15 8.88178420e-15 0.00000000e+00 5.84162772e-05
∑∂S/∂U ⋅ Uₜ : -2.99530351e-15
────────────────────────────────────────────────────────────────────────────────────────────────────

ERROR: LoadError: StackOverflowError:
Stacktrace:
[1] materialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}})
@ StrideArrays ~/.julia/packages/StrideArrays/RDSui/src/broadcast.jl:185
[2] macro expansion
@ ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:530 [inlined]
[3] vmaterialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}}, #unused#::Val{:StrideArrays}, #unused#::Val{(true, 0, 0, 0, true, 4, 32, 15, 64, 0x0000000000000001, 0, true)}, #unused#::Val{((true,), (true,), (true,), (), (), (), ())})
@ LoopVectorization ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:530
[4] vmaterialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}}, #unused#::Val{:StrideArrays}, #unused#::Val{(true, 0, 0, 0, true, 4, 32, 15, 64, 0x0000000000000001, 0, true)})
@ LoopVectorization ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:664
[5] _materialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}}, #unused#::Val{(true, 0, 0, 0, true, 4, 32, 15, 64, 0x0000000000000001, 0, true)})
@ StrideArrays ~/.julia/packages/StrideArrays/RDSui/src/broadcast.jl:178
--- the last 5 lines are repeated 19994 more times ---
[99976] materialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}})
@ StrideArrays ~/.julia/packages/StrideArrays/RDSui/src/broadcast.jl:185
[99977] macro expansion
@ ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:530 [inlined]
[99978] vmaterialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}}, #unused#::Val{:StrideArrays}, #unused#::Val{(true, 0, 0, 0, true, 4, 32, 15, 64, 0x0000000000000001, 0, true)}, #unused#::Val{((true,), (true,), (true,), (), (), (), ())})
@ LoopVectorization ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:530
[99979] vmaterialize!(dest::StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, bc::Base.Broadcast.Broadcasted{StrideArrays.CartesianStyle{Tuple{Int64}, 1}, Nothing, typeof(DiffEqBase.calculate_residuals), Tuple{StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, StrideArraysCore.PtrArray{Tuple{Int64}, (true,), Float64, 1, 1, 0, (1,), Tuple{Static.StaticInt{8}}, Tuple{Static.StaticInt{1}}}, Float64, Float64, Base.RefValue{typeof(DiffEqBase.ODE_DEFAULT_NORM)}, Float64}}, #unused#::Val{:StrideArrays}, #unused#::Val{(true, 0, 0, 0, true, 4, 32, 15, 64, 0x0000000000000001, 0, true)})
@ LoopVectorization ~/.julia/packages/LoopVectorization/DDH6Z/src/broadcast.jl:664
in expression starting at /mnt/ssd/home/efaulhab/git/Trixi.jl/examples/p4est_3d_dgsem/elixir_euler_circular_wind_nonconforming.jl:161
```

This affects all elixirs using `RDPK3SpFSAL49` with threading enabled (`RDPK3SpFSAL49(thread=OrdinaryDiffEq.True())`) and is caused by https://github.com/JuliaSIMD/StrideArrays.jl/issues/62.

Contributor guide

Open the contributing guide

Research direction

Start with examples/p4est_3d_dgsem/elixir_euler_circular_wind_nonconforming.jl at the reported line 161, and reproduce the failure using RDPK3SpFSAL49(thread=OrdinaryDiffEq.True()). Compare the affected elixirs that use this threaded solver and the linked StrideArrays.jl issue; done means those elixirs no longer produce StackOverflowError with threading enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.