astropy / astropy/astropy

TST: rare flakyness (or pollution) in a recently introduced subprocess-based test

Open
#17,775 5 comments 0 reactions 0 assignees View on GitHub
Bug Refactoring testing
Dominant language
Python
Stars
5.3k
Forks
2.2k
Avg merge
1d 20h
Merged PRs (30d)
81

Description

Looking for other sources of test pollution locally, I hit (I assume by chance) a failure mode for this test, introduced in #17598

https://github.com/astropy/astropy/blob/768c5ff1383b9498d2f973663a54305248527b8b/astropy/tests/tests/test_imports.py#L46-L56

It is *extremely* puzzling to me that this test might fail in a subprocess, let alone have anything to do with pollution, but unfortunately it is also very hard to debug; I don't even have a good traceback to work with. All I have is
```
subprocess.CalledProcessError: Command '['/Users/clm/dev/astropy-project/coordinated/astropy/.venv/bin/python3', '-c', "\nimport astropy, sys\nassert 'astropy.units' not in sys.modules\nastropy.units\nassert 'astropy.units' in sys.modules\n"]' returned non-zero exit status 1.
```
I don't have a clear idea how to rewrite this test to at least make it easier to debug when it fails. I tried using `monkeypatch.delitem(sys.modules, "astropy.units", raising=False)`, but it doesn't suffice to simulate "astropy.units has not been imported yet" and the second assert statement fails.

_Originally posted by @neutrinoceros in https://github.com/astropy/astropy/pull/17598#discussion_r1953236865_

ping @mhvk

Contributor guide

Open the contributing guide

Research direction

Start with astropy/tests/tests/test_imports.py, especially the subprocess-based test around lines 46–56, and reproduce the reported CalledProcessError locally. Investigate why the subprocess import-state assertions can fail and whether test pollution is involved. Done means the test reliably checks the intended import behavior and provides enough failure information to debug future failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.