trixi-framework / trixi-framework/Trixi.jl
Parallel p4est simulation crashes when ranks have zero elements
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
Apparently this has been around before, see the great analysis in https://github.com/trixi-framework/Trixi.jl/issues/1096.
However, I am still facing the following issue:
## Example to reproduce
Take
https://github.com/trixi-framework/Trixi.jl/blob/main/examples/p4est_3d_dgsem/elixir_advection_amr.jl
and apply the following changes:
- `initial_refinement_level = 1`
- `tspan = (0.0, 30.0)`
- ```
base_level = 0,
med_level = 1, med_threshold = 10.0,
max_level = 2, max_threshold = 50.0)
```
Run on 2 MPI ranks.
## Error 1
The simulation fails at
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d_parallel.jl#L17
because `get_node_vars` tries to access the first element, which does not exist on one of the two ranks.
This can be avoided by using something like
```julia
l2_error = zero(func(zero(SVector{nvariables(equations), eltype(u)}), equations))
```
## Error 2
Next, the simulation fails at
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d_parallel.jl#L73-L77
because the index of the first element is passed to the anonymous function
https://github.com/trixi-framework/Trixi.jl/blob/fc15c3873d2c0174db1f0a38aab13ff5e6132379/src/callbacks_step/analysis_dg3d.jl#L259-L264
A quick fix was to replace the body by something like
```
if nelements(mesh, dg, cache) == 0
zero(eltype(u))
else
# see above
end
```
Things then work as expected. But how would I solve this properly?
Contributor guide
Assessment
This issue has not been assessed yet.