NCAS-CMS / NCAS-CMS/cf-python

CI: on one workflow, fix installation to make ESMF available

Open
#629 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CI installation testing
Dominant language
Python
Stars
150
Forks
23
Avg merge
1d 11h
Merged PRs (30d)
2

Description

The esmf library isn't being installed properly for our test-source-dist CI workflow, so that it isn't available and therefore regridding capability depending on ESMPy is erroring out, for example as shown from this snippet in the logs from the recent job https://github.com/NCAS-CMS/cf-python/actions/runs/4551782556/jobs/8026279312?pr=627:

2023-03-29T08:26:16.2503387Z Run date: 2023-03-29 08:26:14.821170
2023-03-29T08:26:16.2503879Z Platform: Linux-5.15.0-1034-azure-x86_64-with-glibc2.10 
2023-03-29T08:26:16.2504206Z HDF5 library: 1.12.2 
2023-03-29T08:26:16.2504471Z netcdf library: 4.8.1 
2023-03-29T08:26:16.2505011Z udunits2 library: /usr/share/miniconda3/envs/cf-latest/lib/libudunits2.so.0 
2023-03-29T08:26:16.2505343Z ESMF: not available 
2023-03-29T08:26:16.2505728Z Python: 3.8.16 /usr/share/miniconda3/envs/cf-latest/bin/python
2023-03-29T08:26:16.2506241Z dask: 2023.3.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/dask/__init__.py
2023-03-29T08:26:16.2506817Z netCDF4: 1.6.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/netCDF4/__init__.py
2023-03-29T08:26:16.2507398Z psutil: 5.9.4 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/psutil/__init__.py
2023-03-29T08:26:16.2507988Z packaging: 23.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/packaging/__init__.py
2023-03-29T08:26:16.2509002Z numpy: 1.24.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/numpy/__init__.py
2023-03-29T08:26:16.2509565Z scipy: 1.10.1 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/scipy/__init__.py
2023-03-29T08:26:16.2510160Z matplotlib: 3.2.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/matplotlib/__init__.py
2023-03-29T08:26:16.2510727Z cftime: 1.6.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cftime/__init__.py
2023-03-29T08:26:16.2511303Z cfunits: 3.3.5 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfunits/__init__.py
2023-03-29T08:26:16.2512063Z cfplot: 3.2.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfplot/__init__.py
2023-03-29T08:26:16.2512681Z cfdm: 1.10.0.3 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/__init__.py
2023-03-29T08:26:16.2513175Z cf: 3.14.1 /home/runner/tmp/test_cf-python/cf/__init__.py
2023-03-29T08:26:16.2513593Z 
2023-03-29T08:26:16.2513950Z 
2023-03-29T08:26:16.2514362Z ----------------------------------------------------------------------
2023-03-29T08:26:16.2515213Z Ran 12 tests in 0.958s
2023-03-29T08:26:16.2515424Z 
2023-03-29T08:26:16.2515510Z OK
2023-03-29T08:26:17.7718166Z Traceback (most recent call last):
2023-03-29T08:26:17.7728610Z   File "test_RegridOperator.py", line 10, in <module>
2023-03-29T08:26:17.7729478Z     class RegridOperatorTest(unittest.TestCase):
2023-03-29T08:26:17.7730287Z   File "test_RegridOperator.py", line 13, in RegridOperatorTest
2023-03-29T08:26:17.7731409Z     r = src.regrids(dst, "linear", return_operator=True)
2023-03-29T08:26:17.7732380Z   File "/home/runner/tmp/test_cf-python/cf/decorators.py", line 71, in precede_with_kwarg_deprecation_check
2023-03-29T08:26:17.7732831Z     operation_method_result = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7733518Z   File "/usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
2023-03-29T08:26:17.7734001Z     processed_copy = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7734514Z   File "/home/runner/tmp/test_cf-python/cf/field.py", line 14169, in regrids
2023-03-29T08:26:17.7734812Z     return regrid(
2023-03-29T08:26:17.7735269Z   File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 362, in regrid
2023-03-29T08:26:17.7735642Z     ESMF_manager = ESMF_initialise()  # noqa: F841
2023-03-29T08:26:17.7736172Z   File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 1169, in ESMF_initialise
2023-03-29T08:26:17.7736516Z     raise RuntimeError(
2023-03-29T08:26:17.7736861Z RuntimeError: Regridding will not work unless the ESMF library is installed
2023-03-29T08:26:17.9028758Z + test_result_status=1
2023-03-29T08:26:17.9034545Z + set -x
2023-03-29T08:26:17.9035492Z + exit 1
2023-03-29T08:26:17.9050536Z ##[error]Process completed with exit code 1.

ESMF did get installed OK for this workflow before, so something has changed. I don't have time to look into this before Easter, but note for that job, as an example, from a quick search through the logs, the library does get downgraded, which probably relates to the issue:

2023-03-29T08:21:43.8814896Z The following packages will be SUPERSEDED by a higher-priority channel:
2023-03-29T08:21:43.8815149Z 
2023-03-29T08:21:43.8815609Z   esmpy              conda-forge/noarch::esmpy-8.4.1-pyhc1~ --> conda-forge/linux-64::esmpy-8.4.0-nompi_py38h2b78397_2 
2023-03-29T08:21:43.8815916Z 
2023-03-29T08:21:43.8816072Z The following packages will be DOWNGRADED:
2023-03-29T08:21:43.8816273Z 
2023-03-29T08:21:43.8816563Z   esmf                               8.4.1-nompi_h5b4e5e4_2 --> 8.4.0-nompi_hdb2cfa9_4 
2023-03-29T08:21:43.8817029Z   hdf4                                    4.2.15-h501b40f_6 --> 4.2.15-h9772cbc_5 
2023-03-29T08:21:43.8817524Z   hdf5                            1.14.0-nompi_hb72d44e_103 --> 1.12.2-nompi_h4df4325_101 
2023-03-29T08:21:43.8818006Z   icu                                       72.1-hcb278e6_0 --> 67.1-he1b5a44_0 
2023-03-29T08:21:43.8818447Z   lcms2                                     2.15-haa2dc70_1 --> 2.15-hfd0df8a_0 
2023-03-29T08:21:43.8818920Z   libdeflate                                1.18-h0b41bf4_0 --> 1.17-h0b41bf4_0 
2023-03-29T08:21:43.8819414Z   libnetcdf                        4.9.2-nompi_hf3f8848_102 --> 4.8.1-nompi_h21705cb_104 
2023-03-29T08:21:43.8819901Z   libtiff                                  4.5.0-ha587672_6 --> 4.5.0-h6adf6a1_2 
2023-03-29T08:21:43.8820381Z   matplotlib-base                      3.7.1-py38hd6c3c57_0 --> 3.2.2-py38h5d868c9_1 
2023-03-29T08:21:43.8820920Z   netcdf-fortran                   4.6.0-nompi_h3142581_105 --> 4.6.0-nompi_he1eeb6f_102 
2023-03-29T08:21:43.8821465Z   netcdf4                      1.6.3-nompi_py38hd31e641_102 --> 1.6.0-nompi_py38h6b4b75c_103 
2023-03-29T08:21:43.8822115Z   pillow                               9.4.0-py38h961100d_2 --> 9.4.0-py38hde6dc18_1 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test-source-dist CI workflow and inspect its package-installation steps, focusing on the logged esmpy and esmf downgrade. Use test_RegridOperator.py and its regridding run to verify the workflow environment. Done means ESMF is available in the workflow and the regridding test no longer fails with the missing-library error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.