E3SM-Project / E3SM-Project/scream
Frontier: SCREAM initialization of ne4 on 2 nodes (16 GPUs) has a team size bug.
- Dominant language
- No language data
- Stars
- 79
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
On two nodes of Frontier (16 GPUs total), the following configuration
```
COMPSET=F2010-SCREAMv1
RES=ne4pg2_ne4pg2
```
produces the error
```
Kokkos::TeamPolicy< HIP > the team size is too large. Team size x vector length must be smaller than 1024.
```
Running on one node is fine. I believe this evidence implies that there is a kernel in initialization that is setting the team size to a number that depends on the number of elements in such a way that if the number is too small, the team size is too large.
This issue requires, first, tracking down the responsible kernel(s); second, rewriting it or them. A RangePolicy might be more appropriate than a TeamPolicy.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the F2010-SCREAMv1 configuration with RES=ne4pg2_ne4pg2 on two Frontier nodes and compare it with a one-node run. Trace the initialization kernels that use Kokkos TeamPolicy until the team-size error is identified; done means the two-node, 16-GPU initialization completes successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100