E3SM-Project / E3SM-Project/scream
ML correction not supported on machines outside of quartz and ruby
- Dominant language
- No language data
- Stars
- 79
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
After merging PR #2442, we ran into issues on machines including perlmutter, ascent, and crusher during nightly tests. But we did get through on mappy. We subsequently fixed the issue with PR #2494, which disables compiling ML correction on all machines unless `SCREAM_ENABLE_ML_CORRECTION` is set.
Documenting here the issues and possible solutions:
1. perlmutter: #2501 documents the issue and solution
2. ascent: failed at runtime with `GLIBCXX_3.4.26' not found`. Either try a different version of python (unlikely to work) or append conda libraries to PATH. We decided it was not worth it to investigate more at the moment.
3. crusher: we probably just need to add `set(PYTHON_EXECUTABLE "/opt/cray/pe/python/3.9.13.1/bin/python3" CACHE STRING "" FORCE)` to `gnu_crusher.cmake` and `gnugpu_crusher.cmake`.
None of the above has been implemented because it is unclear how to trigger the nightly test without merging a PR into master and wait for the automatic nightly test.
Additionally, after being able to compile ML correction, we would need to set up a dedicated python virtual environment on each machine for the unit test to work. The venv would include python packages such as tensorflow, xarray, and other custom ML code. Such venv is currently only available on quartz and ruby.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with #2501 and the fix in PR #2494, then inspect gnu_crusher.cmake and gnugpu_crusher.cmake. Determine how nightly tests can be triggered and whether the proposed Crusher Python setting, machine-specific virtual environments, and the documented Perlmutter and Ascent solutions can be validated; done means ML correction compiles and its unit test works on the affected machines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, python, tensorflow
- Domain
- build-system, hpc, machine-learning, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100