CI: on one workflow, fix installation to make ESMF available
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 150
- Forks
- 23
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
The esmf library isn't being installed properly for our test-source-dist CI workflow, so that it isn't available and therefore regridding capability depending on ESMPy is erroring out, for example as shown from this snippet in the logs from the recent job https://github.com/NCAS-CMS/cf-python/actions/runs/4551782556/jobs/8026279312?pr=627:
2023-03-29T08:26:16.2503387Z Run date: 2023-03-29 08:26:14.821170
2023-03-29T08:26:16.2503879Z Platform: Linux-5.15.0-1034-azure-x86_64-with-glibc2.10
2023-03-29T08:26:16.2504206Z HDF5 library: 1.12.2
2023-03-29T08:26:16.2504471Z netcdf library: 4.8.1
2023-03-29T08:26:16.2505011Z udunits2 library: /usr/share/miniconda3/envs/cf-latest/lib/libudunits2.so.0
2023-03-29T08:26:16.2505343Z ESMF: not available
2023-03-29T08:26:16.2505728Z Python: 3.8.16 /usr/share/miniconda3/envs/cf-latest/bin/python
2023-03-29T08:26:16.2506241Z dask: 2023.3.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/dask/__init__.py
2023-03-29T08:26:16.2506817Z netCDF4: 1.6.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/netCDF4/__init__.py
2023-03-29T08:26:16.2507398Z psutil: 5.9.4 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/psutil/__init__.py
2023-03-29T08:26:16.2507988Z packaging: 23.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/packaging/__init__.py
2023-03-29T08:26:16.2509002Z numpy: 1.24.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/numpy/__init__.py
2023-03-29T08:26:16.2509565Z scipy: 1.10.1 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/scipy/__init__.py
2023-03-29T08:26:16.2510160Z matplotlib: 3.2.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/matplotlib/__init__.py
2023-03-29T08:26:16.2510727Z cftime: 1.6.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cftime/__init__.py
2023-03-29T08:26:16.2511303Z cfunits: 3.3.5 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfunits/__init__.py
2023-03-29T08:26:16.2512063Z cfplot: 3.2.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfplot/__init__.py
2023-03-29T08:26:16.2512681Z cfdm: 1.10.0.3 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/__init__.py
2023-03-29T08:26:16.2513175Z cf: 3.14.1 /home/runner/tmp/test_cf-python/cf/__init__.py
2023-03-29T08:26:16.2513593Z
2023-03-29T08:26:16.2513950Z
2023-03-29T08:26:16.2514362Z ----------------------------------------------------------------------
2023-03-29T08:26:16.2515213Z Ran 12 tests in 0.958s
2023-03-29T08:26:16.2515424Z
2023-03-29T08:26:16.2515510Z OK
2023-03-29T08:26:17.7718166Z Traceback (most recent call last):
2023-03-29T08:26:17.7728610Z File "test_RegridOperator.py", line 10, in <module>
2023-03-29T08:26:17.7729478Z class RegridOperatorTest(unittest.TestCase):
2023-03-29T08:26:17.7730287Z File "test_RegridOperator.py", line 13, in RegridOperatorTest
2023-03-29T08:26:17.7731409Z r = src.regrids(dst, "linear", return_operator=True)
2023-03-29T08:26:17.7732380Z File "/home/runner/tmp/test_cf-python/cf/decorators.py", line 71, in precede_with_kwarg_deprecation_check
2023-03-29T08:26:17.7732831Z operation_method_result = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7733518Z File "/usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
2023-03-29T08:26:17.7734001Z processed_copy = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7734514Z File "/home/runner/tmp/test_cf-python/cf/field.py", line 14169, in regrids
2023-03-29T08:26:17.7734812Z return regrid(
2023-03-29T08:26:17.7735269Z File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 362, in regrid
2023-03-29T08:26:17.7735642Z ESMF_manager = ESMF_initialise() # noqa: F841
2023-03-29T08:26:17.7736172Z File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 1169, in ESMF_initialise
2023-03-29T08:26:17.7736516Z raise RuntimeError(
2023-03-29T08:26:17.7736861Z RuntimeError: Regridding will not work unless the ESMF library is installed
2023-03-29T08:26:17.9028758Z + test_result_status=1
2023-03-29T08:26:17.9034545Z + set -x
2023-03-29T08:26:17.9035492Z + exit 1
2023-03-29T08:26:17.9050536Z ##[error]Process completed with exit code 1.
ESMF did get installed OK for this workflow before, so something has changed. I don't have time to look into this before Easter, but note for that job, as an example, from a quick search through the logs, the library does get downgraded, which probably relates to the issue:
2023-03-29T08:21:43.8814896Z The following packages will be SUPERSEDED by a higher-priority channel:
2023-03-29T08:21:43.8815149Z
2023-03-29T08:21:43.8815609Z esmpy conda-forge/noarch::esmpy-8.4.1-pyhc1~ --> conda-forge/linux-64::esmpy-8.4.0-nompi_py38h2b78397_2
2023-03-29T08:21:43.8815916Z
2023-03-29T08:21:43.8816072Z The following packages will be DOWNGRADED:
2023-03-29T08:21:43.8816273Z
2023-03-29T08:21:43.8816563Z esmf 8.4.1-nompi_h5b4e5e4_2 --> 8.4.0-nompi_hdb2cfa9_4
2023-03-29T08:21:43.8817029Z hdf4 4.2.15-h501b40f_6 --> 4.2.15-h9772cbc_5
2023-03-29T08:21:43.8817524Z hdf5 1.14.0-nompi_hb72d44e_103 --> 1.12.2-nompi_h4df4325_101
2023-03-29T08:21:43.8818006Z icu 72.1-hcb278e6_0 --> 67.1-he1b5a44_0
2023-03-29T08:21:43.8818447Z lcms2 2.15-haa2dc70_1 --> 2.15-hfd0df8a_0
2023-03-29T08:21:43.8818920Z libdeflate 1.18-h0b41bf4_0 --> 1.17-h0b41bf4_0
2023-03-29T08:21:43.8819414Z libnetcdf 4.9.2-nompi_hf3f8848_102 --> 4.8.1-nompi_h21705cb_104
2023-03-29T08:21:43.8819901Z libtiff 4.5.0-ha587672_6 --> 4.5.0-h6adf6a1_2
2023-03-29T08:21:43.8820381Z matplotlib-base 3.7.1-py38hd6c3c57_0 --> 3.2.2-py38h5d868c9_1
2023-03-29T08:21:43.8820920Z netcdf-fortran 4.6.0-nompi_h3142581_105 --> 4.6.0-nompi_he1eeb6f_102
2023-03-29T08:21:43.8821465Z netcdf4 1.6.3-nompi_py38hd31e641_102 --> 1.6.0-nompi_py38h6b4b75c_103
2023-03-29T08:21:43.8822115Z pillow 9.4.0-py38h961100d_2 --> 9.4.0-py38hde6dc18_1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the test-source-dist CI workflow and inspect its package-installation steps, focusing on the logged esmpy and esmf downgrade. Use test_RegridOperator.py and its regridding run to verify the workflow environment. Done means ESMF is available in the workflow and the regridding test no longer fails with the missing-library error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100