GenericMappingTools / GenericMappingTools/pygmt
How to resolve flaky tests resulting from using a single GMT session
- Dominant language
- Python
- Stars
- 874
- Forks
- 255
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 40
Description
**Description of the problem**
There's been instances of [flaky](https://docs.pytest.org/en/stable/flaky.html) tests in PyGMT's test suite reported in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-825561365. This likely stems from the fact that PyGMT uses a single GMT session (initiated during `import pygmt`) instead of separate GMT sessions for each figure (see https://github.com/GenericMappingTools/pygmt/pull/327#issuecomment-541782890).
@meghanrjones asked about whether we should stick with using a single GMT session in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-827230998, or use independent sessions per figure
> I understand the original logic behind a single GMT session for all tests in https://github.com/GenericMappingTools/pygmt/pull/327#issuecomment-541782890. Still, I don't expect that users will be attempting to use the entire PyGMT library in a single session, which is the goal of the test suite. So I think it would be worth revisiting this decision. Could it be possible to periodically test the examples/tutorials against baseline images to ensure that producing multiple plots in a single session is consistent and have the unit tests each use individual sessions?
**Full code that generated the error**
Flaky tests are hard to reproduce (that is their definition actually), but in PyGMT's case, can be found e.g. when a single test passing on `pytest pygmt/tests/test_somemodule.py` fails when ran using `make test`, or vice versa.
E.g. as reported by @meghanrjones in https://github.com/GenericMappingTools/pygmt/issues/1217#issuecomment-825753479
> edit: I have not yet been able to figure out a solution. The two makecpt tests fail if there is a docstring example that imports pygmt and instantiates a figure (e.g., `extract_region()` in `pygmt/clib/session.py` and `pygmt/src/grdfilter.py`) and is tested before `pygmt/tests/test_makecpt.py`.
Related issues affected by having a single GMT session:
- #217
- https://github.com/GenericMappingTools/pygmt/issues/372#issuecomment-551855032
- #733
- #1582
**System information**
Please paste the output of `python -c "import pygmt; pygmt.show_versions()"`:
```
PyGMT information:
version: v0.3.2.dev117+g7466dc31
System information:
python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) [GCC 9.3.0]
executable: ~/username/miniconda3/envs/pygmt/bin/python
machine: Linux-5.4.0-72-generic-x86_64-with-debian-bullseye-sid
Dependency information:
numpy: 1.17.1
pandas: 1.2.3
xarray: 0.17.0
netCDF4: 1.5.6
packaging: 20.9
ghostscript: 9.53.3
gmt: 6.2.0rc1
GMT library information:
binary dir: ~/username/miniconda3/envs/pygmt/bin
cores: 6
grid layout: rows
library path: ~/username/miniconda3/envs/pygmt/lib/libgmt.so
padding: 2
plugin dir: ~/username/miniconda3/envs/pygmt/lib/gmt/plugins
share dir: ~/username/miniconda3/envs/pygmt/share/gmt
version: 6.2.0rc1
```
Contributor guide
Assessment
This issue has not been assessed yet.