E3SM-Project / E3SM-Project/scream

ML correction not supported on machines outside of quartz and ruby

Open
#2,509 0 comments 0 reactions 0 assignees View on GitHub
machine learning priority:low
Dominant language
No language data
Stars
79
Forks
54
PR merge metrics
No merged PRs in 30d

Description

After merging PR #2442, we ran into issues on machines including perlmutter, ascent, and crusher during nightly tests. But we did get through on mappy. We subsequently fixed the issue with PR #2494, which disables compiling ML correction on all machines unless `SCREAM_ENABLE_ML_CORRECTION` is set.

Documenting here the issues and possible solutions:
1. perlmutter: #2501 documents the issue and solution
2. ascent: failed at runtime with `GLIBCXX_3.4.26' not found`. Either try a different version of python (unlikely to work) or append conda libraries to PATH. We decided it was not worth it to investigate more at the moment.
3. crusher: we probably just need to add `set(PYTHON_EXECUTABLE "/opt/cray/pe/python/3.9.13.1/bin/python3" CACHE STRING "" FORCE)` to `gnu_crusher.cmake` and `gnugpu_crusher.cmake`.

None of the above has been implemented because it is unclear how to trigger the nightly test without merging a PR into master and wait for the automatic nightly test.

Additionally, after being able to compile ML correction, we would need to set up a dedicated python virtual environment on each machine for the unit test to work. The venv would include python packages such as tensorflow, xarray, and other custom ML code. Such venv is currently only available on quartz and ruby.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with #2501 and the fix in PR #2494, then inspect gnu_crusher.cmake and gnugpu_crusher.cmake. Determine how nightly tests can be triggered and whether the proposed Crusher Python setting, machine-specific virtual environments, and the documented Perlmutter and Ascent solutions can be validated; done means ML correction compiles and its unit test works on the affected machines.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, python, tensorflow
Domain
build-system, hpc, machine-learning, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.