trixi-framework / trixi-framework/Trixi.jl
Boundary flux in the `StructuredMesh` solver doesn't scale well on multiple threads
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
Especially in 3D simulations with the compressible Euler equations (like `examples/structured_3d_dgsem/elixir_euler_source_terms_nonperiodic_curved.jl`), the boundary flux quickly becomes the most expensive part of the simulation when using multiple threads.
Right now, the corresponding code (`calc_boundary_flux!` in `src/solvers/dgsem_structured/dg_3d.jl`) does not contain any `@threaded` annotations.
I tried adding `@threaded` to the loops, and I tried rewriting them to not have nested loops, something like:
```julia
@threaded for cell in Iterators.product(axes(mesh, 3), axes(mesh, 2))
cell_z, cell_y = cell
```
Both didn't fix the problem completely, so this problem may require a bit more work and some restructuring of data structures.
Contributor guide
Research direction
Run the 3D Euler example at examples/structured_3d_dgsem/elixir_euler_source_terms_nonperiodic_curved.jl and inspect calc_boundary_flux! in src/solvers/dgsem_structured/dg_3d.jl. Compare threaded loop variants and measure boundary-flux scaling across multiple threads. Done means the boundary flux no longer dominates as quickly and scaling improves for the described 3D simulations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100