NCAS-CMS / NCAS-CMS/cf-python

Remove test suite outcome dependence on run directory

Aperta
#97 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

testing
Lingua principale
Python
Stelle
150
Fork
23
Merge medio
1g 11h
PR unite (30g)
2

Descrizione

At the moment the pass/fail outcome of the test suite depends on the directory which the suite script run_test.py is run from. That is, certain test modules & hence the full suite seem to have a dependence on the initial run directory.

As a demonstration, if the full test suite passes when run from the cf-python/cf/test repo where the tests live, it will fail when run from some outside directory, but not in a consistent way given the specific outside directory. For example, from the root directory repo I observe that the unit test loading errors:

$ pwd
/home/sadie/cf-python
$ python cf/test/run_tests.py 
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:55:20.056541
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie/cf-python
cf.abstract (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
cf.abstract (unittest.loader._FailedTest) ... ERROR
test_create_field (cf.test.setup_create_field.create_fieldTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 2 tests in 0.456s

FAILED (errors=1)
test_CellMeasure_identity (cf.test.test_CellMeasure.CellMeasureTest) ... ok
...
...
...
test_manage_log_level_via_verbose_attr (cf.test.test_decorators.DecoratorsTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 275 tests in 651.759s

FAILED (errors=1)

& outside of the repo directory completely the run_tests.py script picks up on & runs zero tests, notably implying a pass when this may not be the case. For instance, from my $HOME:

$ pwd
/home/sadie
$ ls ~/cf-python/cf/test/run_tests.py 
/home/sadie/cf-python/cf/test/run_tests.py
$ python cf-python/cf/test/run_tests.py
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:26:43.800432
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

We should remove all coupling between run directory and test module outcome.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia riproducendo i risultati differenti dal repository, dalla sua root e da $HOME usando cf/test/run_tests.py. Esamina la discovery dei test e la gestione dei percorsi in quello script, quindi verifica che venga selezionata la stessa suite indipendentemente dalla directory di lavoro. Il lavoro è completato quando i test non falliscono a causa della directory di avvio e una discovery vuota non può segnalare un successo fuorviante.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
testing-qa
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.