ESMCI / ESMCI/cime

Add possibility to use pre-built shared libs during case.build

Open
#4,308 16 comments 0 reactions 2 assignees Claimed by @jasonb5 View on GitHub
Low Priority
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

The need for prebuilt libs stems from a couple of needs:

1. Large nightly testing might shave quite a few minutes if using pre-built shared libs. Yes, the testsuites can share the share libs, but up to a certain extent.
2. We would like to link a TPL to the model. This TPL should be prebuilt (very expensive to build, and/or very complicated configuration), and could benefit from using, say, pio or gptl. We would like to use _the same version_ of pio/gptl that e3sm would link against.

Here's how I envision this to be implemented:
- Add an XML var `USE_PREBUILT_LIBS` (or something like that), with valid values `none`, `all`, or a comma-separated list of libs (e.g, `USE_PREBUILT_LIBS=pio,gptl`).
- In case.build, check this var, and only build the libs that are not to be used as prebuilt.
- In config_machines, set vars like `${LIBNAME}_ROOT`. Then, case.build will add `$compiler/$buildtype` to get the full installation path (where `buildtype` is either `debug` or `release`). If we feel brave, we could also add some more buildtypes, based on other XML config options (I'm thinking about stuff like certain PIO compile-time options).

Additional comments:
- `create_test` could default to `USE_PREBUILT_LIBS="none"`, for bwd compatibility. Or we could have config_machines store a default for this variable, so that e3sm/cesm can choose different approaches, or we could have different approaches on different machines. E.g., on new machines we start with "none", and if the maintainer finds time, they can install prebuilt libs.
- `create_test` could have a flag, like `--use-prebuilt-libs=a,b,c` to force using prebuilt libs for `a,b,c`, if the machine default is to build them on the fly. A symmetric option could be `--build-shared=d,e`, in case the machine default is do use pre-built. Obviously, using both at the same time is not allowed.
- Compilers change, so we could store a compiler version sha along the installation. When case.build runs, it can check the current compiler version against the stored sha, and, if they don't match, build the lib on the fly, maybe printing a big warning "WARNING, installation for lib X, compiler Y, build type Z is expired. Contact machine maintainer".
- I argue we don't need to have 1 installation for each flag/options configuration. If the user wants fine grain control over shared libs flags, they can have the libs built on the fly. Prebuilt libs are for users that are ok with some sort of default settings.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.