trixi-framework / trixi-framework/Trixi.jl
Performance issues on RAMSES
- Dominant language
- Julia
- Stars
- 731
- Forks
- 167
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
When running Trixi.jl on the UoC's cluster RAMSES, I noticed some performance issues.
I ran
```julia
include("examples/tree_3d_dgsem/elixir_euler_taylor_green_vortex.jl")
```
on this RAMSES node:
```
CPU(s): 2 x AMD EPYC 9654 96-Core Processor
CPU target: znver4
Cores: 192 (192 CPU-threads)
NUMA domains: 2 (96 cores each)
```
and on our old workstation:
```
CPU(s): 1 x AMD Ryzen Threadripper 3990X 64-Core Processor
CPU target: znver2
Cores: 64 (128 CPU-threads due to 2-way SMT)
NUMA domains: 1 (64 cores each)
```
On RAMSES:
```
Tot / % measured: 32.5s / 79.3% 135MiB / 64.0%
Section ncalls time %tot avg alloc %tot avg
────────────────────────────────────────────────────────────────────────────────────
rhs! 5.99k 24.7s 95.9% 4.12ms 8.78MiB 10.1% 1.50KiB
reset ∂u/∂t 5.99k 4.26s 16.5% 710μs 0.00B 0.0% 0.00B
volume integral 5.99k 4.18s 16.2% 698μs 2.10MiB 2.4% 368B
interface flux 5.99k 4.12s 16.0% 688μs 1.92MiB 2.2% 336B
prolong2interfaces 5.99k 4.03s 15.6% 672μs 1.74MiB 2.0% 304B
surface integral 5.99k 3.95s 15.4% 660μs 1.65MiB 1.9% 288B
Jacobian 5.99k 3.94s 15.3% 658μs 1.37MiB 1.6% 240B
```
on the Threadripper:
```
Tot / % measured: 6.18s / 48.5% 59.2MiB / 99.2%
Section ncalls time %tot avg alloc %tot avg
────────────────────────────────────────────────────────────────────────────────────
rhs! 5.99k 2.40s 79.8% 400μs 8.78MiB 15.0% 1.50KiB
volume integral 5.99k 728ms 24.2% 121μs 2.10MiB 3.6% 368B
reset ∂u/∂t 5.99k 475ms 15.8% 79.2μs 0.00B 0.0% 0.00B
interface flux 5.99k 392ms 13.1% 65.4μs 1.92MiB 3.3% 336B
~rhs!~ 5.99k 280ms 9.3% 46.8μs 4.78KiB 0.0% 0.82B
prolong2interfaces 5.99k 257ms 8.6% 42.9μs 1.74MiB 3.0% 304B
surface integral 5.99k 189ms 6.3% 31.5μs 1.65MiB 2.8% 288B
Jacobian 5.99k 73.3ms 2.4% 12.2μs 1.37MiB 2.3% 240B
```
I noticed that the time interval between alive outputs varies a lot on the cluster, which is probably related to the problem.
I don't have time to investigate, but there are others with access to this system, e.g. @benegee.
Contributor guide
Research direction
Start by reproducing the Julia command in examples/tree_3d_dgsem/elixir_euler_taylor_green_vortex.jl on RAMSES and the Threadripper setup described in the issue. Compare the reported rhs! profiling sections and the varying interval between alive outputs. Done means identifying the cause of the cluster slowdown and establishing a measurable improvement or a clear diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100