trixi-framework / trixi-framework/Trixi.jl
GPU porting
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
Rough guideline on making parts of Trixi.jl GPU-compatible:
https://github.com/trixi-framework/Trixi.jl/blob/main/docs/src/heterogeneous.md#writing-gpu-kernels
## Discussion
- It would be much nicer to hide away the implementation and launch of kernels, see https://github.com/trixi-framework/TrixiBase.jl/pull/66
- ~~Pass `typeof(mesh)` to kernels, instead of just `mesh`~~ ✅
- Pass `typeof(dg)` to kernels, instead of just `dg`?
- How to deal with `cache`?
## Step 1: Minimal feature set
- [x] Basic solution for the "no cache on GPU" problem
- [x] Basic P4est functionality (2D & 3D): #2590
## Step 2a: "Full" implementation without host-device memory transfers
- [ ] Tutorial on how to use GPU backend (with different precisions)
- [x] Output information on GPU usage in `SummaryCallback` #2906
- [x] `StepsizeCallback`, in particular reduction of speeds #2823
- [x] `AnalysisCallback` #2882
- [ ] `SaveSolutionCallback`
- [ ] `SaveRestartCallback`
## Step 2b: Initial performance analysis & optimization
- [x] More fine-grained parallelization for flux differencing kernel (https://github.com/trixi-framework/Trixi.jl/pull/2590#discussion_r2986542162)
- [ ] 2D `calc_volume_integral!` optimized kernel on quadrature nodes for conservative systems
- [x] 3D `calc_volume_integral!` optimized kernel on quadrature nodes for conservative systems(#3015)
- [ ] 2D `calc_volume_integral!` optimized kernel on quadrature nodes for nonconservative systems
- [x] 3D `calc_volume_integral!` optimized kernel on quadrature nodes for nonconservative systems (#3065)
- [x] `apply_jacobian!` 2D kernel on quadrature nodes (#3013)
- [x] `apply_jacobian!` 3D kernel on quadrature nodes (#3017)
- [x] ` calc_source_terms!` 2D and 3D kernels on quadrature nodes (#3012)
- [x] 2D kernels for `prolong2boundaries!` and `calc_boundary_flux!` (started in #3007, moved to #3077)
- [ ] The 2D `prolong2boundaries_per_boundary!` in `dg_2d_gpu.jl` needs to be moved in the CPU file `dgsem_p4est/dg_2d.jl`, see [comment regarding 2D kernel](https://github.com/trixi-framework/Trixi.jl/pull/3034#discussion_r3330725479))
- [x] 3D kernels for `prolong2boundaries!` and `calc_boundary_flux!` (#3034)
- [x] Analysis and eventual optimization of `calc_surface_integral!` and `prolong2interfaces`
- [x] Fuse `prolong2interfaces` and `calc_interface_flux!` (#3077)
- [x] Fuse looped may be advantageous in some cases (for example `calc_source_terms!` and `apply_jacobian!` can be fused)
- [x] 2D and 3D optimized kernels for `prolong2boundaries` and `calc_boundary_flux!` (see [Avoid launching a kernel for each boundary](https://github.com/trixi-framework/Trixi.jl/pull/3034#pullrequestreview-4390002985, #3207)
- [x] 2D and 3D optimization for `calc_surface_integrals!` (#3058)
- [x] 3D GPU volume kernel with LDS (#3229, #3206)
- [ ] 2D GPU volume kernel with LDS
## Step 3: Advanced features
- [ ] Proper solution for the "no cache on GPU" problem
- [ ] MPI / multi-GPU
- [ ] Get rid of `VecOfArrays` #2952
- [ ] AMR (started in #3101)
- [x] Fix "undefined external symbol 'log'" (see [comment](https://github.com/trixi-framework/Trixi.jl/pull/2590/changes#diff-8dd46bd0d7489bd62ba258deebb17e7a73e1d0e980aa2ada2530e3496b51a1c0R29-R30))
- [ ] Figure out whether `backend` should be first argument to functions, or last, or ...
## Later
- [x] Figure out how to show backend in splash screen #2906
- [ ] Unify/simplify `mesh` vs. `MeshT`
- [ ] T8code
- [ ] Clean up code paths, e.g., consider getting rid of unused `interfaces.u` array (https://github.com/trixi-framework/Trixi.jl/pull/3077#discussion_r3702642427).
- [ ] restart option compatibility
## Not planned
- TreeMesh
- StructuredMesh
- UnstructuredMesh
## Temporary roadmap (by @MarcoArtiano)
1. ~~3D GPU volume kernel with LDS~~ (#3206)
2. ~~3D GPU turbo volume kernel with LDS using the `FluxTurbo` infrastructure~~ (#3229)
3. ~~Fuse Jacobian and source terms~~ (done in #3180)
4. ~~remove `reset_du!`~~ (#3230)
5. Bring everything also to 2D.
Contributor guide
Assessment
This issue has not been assessed yet.