ESCOMP / ESCOMP/CAM-SIMA

Use real GPU snapshot for nvhpc regression test

Open
#469 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
9
Forks
22
Avg merge
8d 5h
Merged PRs (30d)
4

Description

### Issue Type

Code Clean-up

### Issue Description

As referenced in https://github.com/ESCOMP/CAM-SIMA/pull/451, the RRTMGP GPU test is running against a CPU snapshot because:
1. I (Courtney) could not get a GPU snapshot to run with no answer changes
Answer differences for the first timestep using the ne30 GPU snapshot (possible configuration issues present):
```
********** Physics Check Data Results **********

TIMESTEP: 1

Variable # Diffs Max Diff Max Diff loc (rank, col, lev)
-------- ------- -------- -----------------------------
air_temperature 2818800 0.83E-02 (43,131,58)
diffuse_solar_radiative_flux_at_surface_ge_700nm_to_coupler
24300 0.25E+03 (44,66)
diffuse_solar_radiative_flux_at_surface_lt_700nm_to_coupler
24300 0.32E+03 (47,37)
direct_solar_radiative_flux_at_surface_ge_700nm_to_coupler
24288 0.54E+03 (42,339)
direct_solar_radiative_flux_at_surface_lt_700nm_to_coupler
24288 0.51E+03 (47,333)
geopotential_height_wrt_surface 2818800 0.83E-02 (43,131,58)
geopotential_height_wrt_surface_at_interface 2818800 0.83E-02 (43,131,58)
longwave_net_outgoing_flux_at_model_top 48600 0.35E+03 (23,149)
longwave_net_upward_flux_at_surface 48600 0.25E+03 (23,149)
shortwave_net_outgoing_flux_at_model_top 24300 0.13E+04 (47,312)
shortwave_net_upward_flux_at_surface 24300 0.11E+04 (46,181)
tendency_of_air_temperature_due_to_model_physics 2818800 0.78E+02 (24,53,58)
tendency_of_dry_air_enthalpy_at_constant_pressure_due_to_longwave_radiation_adjusted_by_air_pressure_thickness
2818800 0.11E+04 (68,342,34)
tendency_of_dry_air_enthalpy_at_constant_pressure_due_to_shortwave_radiation_adjusted_by_air_pressure_thickness
1409400 0.70E+03 (43,139,27)

********** End Physics Check Data Results **********

```
ALSO the second timestep failed with:
```
cam_ccpp_physics_run: gas_optics(): array play has values outside range
```
2. The only GPU snapshot I was able to make was with ne30 and was thus huge and I felt like it took up too much space on glade without actually working.
3. I'd expect answer differences with the CPU snapshot, but not the massive answer differences that we're seeing:
```
********** Physics Check Data Results **********

TIMESTEP: 1

Variable # Diffs Max Diff Max Diff loc (rank, col, lev)
-------- ------- -------- -----------------------------
air_temperature 28188 0.59E-02 (37,1,58)
diffuse_solar_radiative_flux_at_surface_ge_700nm_to_coupler
242 0.20E+03 (42,2)
diffuse_solar_radiative_flux_at_surface_lt_700nm_to_coupler
242 0.26E+03 (42,2)
direct_solar_radiative_flux_at_surface_ge_700nm_to_coupler
242 0.49E+03 (50,2)
direct_solar_radiative_flux_at_surface_lt_700nm_to_coupler
242 0.50E+03 (50,2)
geopotential_height_wrt_surface 28188 0.59E-02 (37,1,58)
geopotential_height_wrt_surface_at_interface 28188 0.59E-02 (37,1,58)
longwave_net_outgoing_flux_at_model_top 486 0.21E+06 (30,2)
longwave_net_upward_flux_at_surface 486 0.82E+07 (30,2)
shortwave_net_outgoing_flux_at_model_top 242 0.13E+04 (48,3)
shortwave_net_upward_flux_at_surface 242 0.10E+04 (50,2)
tendency_of_air_temperature_due_to_model_physics 28188 0.15E+02 (57,2,1)
tendency_of_dry_air_enthalpy_at_constant_pressure_due_to_longwave_radiation_adjusted_by_air_pressure_thickness
28188 0.54E+07 (25,3,39)
tendency_of_dry_air_enthalpy_at_constant_pressure_due_to_shortwave_radiation_adjusted_by_air_pressure_thickness
14036 0.48E+03 (42,1,33)
```

TLDR; something is wrong with the way the GPU test is running for it to end up with these massive differences (or errors from RRTMGP's internal checks)

This fix may change answers for NVHPC RRTMGP.

### Will this change answers?

Yes

### Will you be implementing this yourself?

Any Software Engineer can do this

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.