NCAS-CMS / NCAS-CMS/cf-python

Remove test suite outcome dependence on run directory

Offen
#97 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

testing
Vorherrschende Sprache
Python
Sterne
150
Forks
23
Ø Merge
1 T. 11 Std.
Gemergte PRs (30 T.)
2

Beschreibung

At the moment the pass/fail outcome of the test suite depends on the directory which the suite script run_test.py is run from. That is, certain test modules & hence the full suite seem to have a dependence on the initial run directory.

As a demonstration, if the full test suite passes when run from the cf-python/cf/test repo where the tests live, it will fail when run from some outside directory, but not in a consistent way given the specific outside directory. For example, from the root directory repo I observe that the unit test loading errors:

$ pwd
/home/sadie/cf-python
$ python cf/test/run_tests.py 
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:55:20.056541
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie/cf-python
cf.abstract (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
cf.abstract (unittest.loader._FailedTest) ... ERROR
test_create_field (cf.test.setup_create_field.create_fieldTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 2 tests in 0.456s

FAILED (errors=1)
test_CellMeasure_identity (cf.test.test_CellMeasure.CellMeasureTest) ... ok
...
...
...
test_manage_log_level_via_verbose_attr (cf.test.test_decorators.DecoratorsTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 275 tests in 651.759s

FAILED (errors=1)

& outside of the repo directory completely the run_tests.py script picks up on & runs zero tests, notably implying a pass when this may not be the case. For instance, from my $HOME:

$ pwd
/home/sadie
$ ls ~/cf-python/cf/test/run_tests.py 
/home/sadie/cf-python/cf/test/run_tests.py
$ python cf-python/cf/test/run_tests.py
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:26:43.800432
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

We should remove all coupling between run directory and test module outcome.

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne damit, die unterschiedlichen Ergebnisse aus dem Repository, dessen Root und $HOME mit cf/test/run_tests.py zu reproduzieren. Untersuche die Testermittlung und die Pfadbehandlung in diesem Skript und verifiziere anschließend, dass unabhängig vom Arbeitsverzeichnis dieselbe Testsuite ausgewählt wird. Erledigt bedeutet, dass Tests nicht aufgrund des Startverzeichnisses fehlschlagen und eine leere Testermittlung keinen irreführenden Erfolg melden kann.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
testing-qa
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.