trixi-framework / trixi-framework/Trixi.jl

Boundary flux in the `StructuredMesh` solver doesn't scale well on multiple threads

Open
#1,000 0 comments 0 reactions 0 assignees View on GitHub
parallelization performance
Dominant language
Julia
Stars
731
Forks
166
Avg merge
2d 18h
Merged PRs (30d)
25

Description

Especially in 3D simulations with the compressible Euler equations (like `examples/structured_3d_dgsem/elixir_euler_source_terms_nonperiodic_curved.jl`), the boundary flux quickly becomes the most expensive part of the simulation when using multiple threads.

Right now, the corresponding code (`calc_boundary_flux!` in `src/solvers/dgsem_structured/dg_3d.jl`) does not contain any `@threaded` annotations.
I tried adding `@threaded` to the loops, and I tried rewriting them to not have nested loops, something like:
```julia
@threaded for cell in Iterators.product(axes(mesh, 3), axes(mesh, 2))
cell_z, cell_y = cell
```
Both didn't fix the problem completely, so this problem may require a bit more work and some restructuring of data structures.

Contributor guide

Open the contributing guide

Research direction

Run the 3D Euler example at examples/structured_3d_dgsem/elixir_euler_source_terms_nonperiodic_curved.jl and inspect calc_boundary_flux! in src/solvers/dgsem_structured/dg_3d.jl. Compare threaded loop variants and measure boundary-flux scaling across multiple threads. Done means the boundary flux no longer dominates as quickly and scaling improves for the described 3D simulations.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.