E3SM-Project / E3SM-Project/scream

Frontier: SCREAM initialization of ne4 on 2 nodes (16 GPUs) has a team size bug.

Open
#2,661 0 comments 0 reactions 0 assignees View on GitHub
bug Frontier GPU
Dominant language
No language data
Stars
79
Forks
54
PR merge metrics
No merged PRs in 30d

Description

On two nodes of Frontier (16 GPUs total), the following configuration
```
COMPSET=F2010-SCREAMv1
RES=ne4pg2_ne4pg2
```
produces the error
```
Kokkos::TeamPolicy< HIP > the team size is too large. Team size x vector length must be smaller than 1024.
```
Running on one node is fine. I believe this evidence implies that there is a kernel in initialization that is setting the team size to a number that depends on the number of elements in such a way that if the number is too small, the team size is too large.

This issue requires, first, tracking down the responsible kernel(s); second, rewriting it or them. A RangePolicy might be more appropriate than a TeamPolicy.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the F2010-SCREAMv1 configuration with RES=ne4pg2_ne4pg2 on two Frontier nodes and compare it with a one-node run. Trace the initialization kernels that use Kokkos TeamPolicy until the team-size error is identified; done means the two-node, 16-GPU initialization completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.