MetOffice / MetOffice/CDDS

`Segmentation fault` from CMIP7 mip_convert functional tests

Open
#832 0 comments 0 reactions 0 assignees View on GitHub
Investigation quality assurance
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

As part of this issue https://github.com/MetOffice/CDDS/issues/802 https://github.com/MetOffice/CDDS/pull/815 a CMIP7 functional test was added (`test_cmip7_mon_mcd_tavg-alh-hxy-u.py`). After introduction of this test, running:
`pytest CDDS/mip_convert/mip_convert/tests/test_functional/test_functional_cmip7 -m slow` causes all tests to pass. However, after the tests have passed there is a segmentation fault e.g.:
```
================================================= 4 passed, 5 warnings in 25.23s ==================================================
Segmentation fault (core dumped)
```

Me and Jared did some investigation into this. Couldn't find a solution but here's a few things that were found:
- It only seems to occur when running the tests locally. When run via spice this doesn't seem to occur.
- Changing the name of test_cmip7_mon_mcd_tavg-alh-hxy-u.py` to `test_cmip7_zmon_mcd_tavg-alh-hxy-u.py` to make this test run last makes the segmentation fault disappear.
- When tracemalloc is used to check for memory usage on that group of cmip7 tests, all of them have `Current memory usage: 0.00 KB; Peak: 0.00 KB` other than the test i added `test_cmip7_mon_mcd_tavg-alh-hxy-u.py` which shows `Current memory usage: 146.08 KB; Peak: 458.97 KB`. So i imagine it's related to code i added in the above issue.
- Attaching tracemalloc to the method called (_[remove_alevhalf_bounds](https://github.com/MetOffice/CDDS/blob/9b7c3e1e3e692604be8fe08c40c7e5887a80b656/mip_convert/mip_convert/new_variable.py#L371)) in the new functional test i added showed dask with the highest mem usage:
`/dask/array/core.py:1283: size=17.9 MiB, count=2, average=9180 KiB`.
- Running tracemalloc on another test showed this as the highest mem usage:
`iris/fileformats/pp.py:1880: size=30.2 MiB, count=987221, average=32 B`
- Using valgrind showed references to cftime and numpy, not sure if these are related or not.
- I think Jared said it could be an issue to do with cmor and persisted state.

## As the tests still pass and it's not causing any errors when run via spice, not going to continue investigating this for the time being. This issue is for a low priority investigation at some point as to what could be causing the memory leaks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running `pytest CDDS/mip_convert/mip_convert/tests/test_functional/test_functional_cmip7 -m slow` and compare behavior when `test_cmip7_mon_mcd_tavg-alh-hxy-u.py` runs last. Inspect `_remove_alevhalf_bounds` in `mip_convert/new_variable.py`, then use tracemalloc and valgrind as described to identify the cause of the post-test segmentation fault or memory leak; done means documenting or fixing the cause without breaking the passing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.