NCAS-CMS / NCAS-CMS/cf-python

Remove test suite outcome dependence on run directory

オープン
#97 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

testing
主要言語
Python
スター
150
フォーク
23
平均マージ
1日 11時間
マージ済み PR(30日)
2

説明

At the moment the pass/fail outcome of the test suite depends on the directory which the suite script run_test.py is run from. That is, certain test modules & hence the full suite seem to have a dependence on the initial run directory.

As a demonstration, if the full test suite passes when run from the cf-python/cf/test repo where the tests live, it will fail when run from some outside directory, but not in a consistent way given the specific outside directory. For example, from the root directory repo I observe that the unit test loading errors:

$ pwd
/home/sadie/cf-python
$ python cf/test/run_tests.py 
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:55:20.056541
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie/cf-python
cf.abstract (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
cf.abstract (unittest.loader._FailedTest) ... ERROR
test_create_field (cf.test.setup_create_field.create_fieldTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 2 tests in 0.456s

FAILED (errors=1)
test_CellMeasure_identity (cf.test.test_CellMeasure.CellMeasureTest) ... ok
...
...
...
test_manage_log_level_via_verbose_attr (cf.test.test_decorators.DecoratorsTest) ... ok

======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
    from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)


----------------------------------------------------------------------
Ran 275 tests in 651.759s

FAILED (errors=1)

& outside of the repo directory completely the run_tests.py script picks up on & runs zero tests, notably implying a pass when this may not be the case. For instance, from my $HOME:

$ pwd
/home/sadie
$ ls ~/cf-python/cf/test/run_tests.py 
/home/sadie/cf-python/cf/test/run_tests.py
$ python cf-python/cf/test/run_tests.py
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:26:43.800432
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py

Running tests from /home/sadie

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

We should remove all coupling between run directory and test module outcome.

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

まず、cf/test/run_tests.py を使って、リポジトリ、そのルート、および $HOME から実行した場合の異なる結果を再現します。そのスクリプトでのテスト検出とパス処理を調べ、次に、作業ディレクトリにかかわらず同じテストスイートが選択されることを確認します。完了の条件は、起動ディレクトリが原因でテストが失敗せず、空の検出結果が誤解を招く成功を報告できないことです。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
testing-qa
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。