trixi-framework / trixi-framework/Trixi.jl

Parallel p4est simulation crashes when ranks have zero elements

Open
#1,973 3 comments 0 reactions 0 assignees View on GitHub
bug parallelization question
Dominant language
Julia
Stars
731
Forks
166
Avg merge
2d 18h
Merged PRs (30d)
25

Description

Apparently this has been around before, see the great analysis in https://github.com/trixi-framework/Trixi.jl/issues/1096.
However, I am still facing the following issue:

## Example to reproduce
Take
https://github.com/trixi-framework/Trixi.jl/blob/main/examples/p4est_3d_dgsem/elixir_advection_amr.jl
and apply the following changes:
- `initial_refinement_level = 1`
- `tspan = (0.0, 30.0)`
- ```
base_level = 0,
med_level = 1, med_threshold = 10.0,
max_level = 2, max_threshold = 50.0)
```
Run on 2 MPI ranks.

## Error 1
The simulation fails at
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d_parallel.jl#L17
because `get_node_vars` tries to access the first element, which does not exist on one of the two ranks.

This can be avoided by using something like
```julia
l2_error = zero(func(zero(SVector{nvariables(equations), eltype(u)}), equations))
```

## Error 2
Next, the simulation fails at
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d_parallel.jl#L73-L77
because the index of the first element is passed to the anonymous function
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d.jl#L259-L264

A quick fix was to replace the body by something like
```
if nelements(mesh, dg, cache) == 0
zero(eltype(u))
else
# see above
end
```

Things then work as expected. But how would I solve this properly?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.