Remove test suite outcome dependence on run directory
まだ誰も着手していません。
- 主要言語
- Python
- スター
- 150
- フォーク
- 23
- 平均マージ
- 1日 11時間
- マージ済み PR(30日)
- 2
説明
At the moment the pass/fail outcome of the test suite depends on the directory which the suite script run_test.py is run from. That is, certain test modules & hence the full suite seem to have a dependence on the initial run directory.
As a demonstration, if the full test suite passes when run from the cf-python/cf/test repo where the tests live, it will fail when run from some outside directory, but not in a consistent way given the specific outside directory. For example, from the root directory repo I observe that the unit test loading errors:
$ pwd
/home/sadie/cf-python
$ python cf/test/run_tests.py
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:55:20.056541
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py
Running tests from /home/sadie/cf-python
cf.abstract (unittest.loader._FailedTest) ... ERROR
======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
package = self._get_module_from_name(name)
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (errors=1)
cf.abstract (unittest.loader._FailedTest) ... ERROR
test_create_field (cf.test.setup_create_field.create_fieldTest) ... ok
======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
package = self._get_module_from_name(name)
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)
----------------------------------------------------------------------
Ran 2 tests in 0.456s
FAILED (errors=1)
test_CellMeasure_identity (cf.test.test_CellMeasure.CellMeasureTest) ... ok
...
...
...
test_manage_log_level_via_verbose_attr (cf.test.test_decorators.DecoratorsTest) ... ok
======================================================================
ERROR: cf.abstract (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: cf.abstract
Traceback (most recent call last):
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 470, in _find_test_path
package = self._get_module_from_name(name)
File "/home/sadie/anaconda3/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
File "/home/sadie/cf-python/cf/abstract/__init__.py", line 1, in <module>
from .coordinate import Coordinate
ImportError: cannot import name 'Coordinate' from 'cf.abstract.coordinate' (/home/sadie/cf-python/cf/abstract/coordinate.py)
----------------------------------------------------------------------
Ran 275 tests in 651.759s
FAILED (errors=1)
& outside of the repo directory completely the run_tests.py script picks up on & runs zero tests, notably implying a pass when this may not be the case. For instance, from my $HOME:
$ pwd
/home/sadie
$ ls ~/cf-python/cf/test/run_tests.py
/home/sadie/cf-python/cf/test/run_tests.py
$ python cf-python/cf/test/run_tests.py
--------------------
CF-PYTHON TEST SUITE
--------------------
Run date: 2020-07-17 01:26:43.800432
Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
HDF5 library: 1.10.4
netcdf library: 4.6.3
udunits2 library: /home/sadie/anaconda3/lib/libudunits2.so.0
python: 3.7.6 /home/sadie/anaconda3/bin/python
netCDF4: 1.5.3 /home/sadie/anaconda3/lib/python3.7/site-packages/netCDF4/__init__.py
cftime: 1.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/cftime/__init__.py
numpy: 1.18.1 /home/sadie/anaconda3/lib/python3.7/site-packages/numpy/__init__.py
psutil: 5.7.0 /home/sadie/anaconda3/lib/python3.7/site-packages/psutil/__init__.py
scipy: 1.4.1 /home/sadie/anaconda3/lib/python3.7/site-packages/scipy/__init__.py
matplotlib: 3.1.3 /home/sadie/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py
ESMF: 8.0.1 /home/sadie/anaconda3/lib/python3.7/site-packages/ESMF/__init__.py
cfdm: 1.8.6 /home/sadie/cfdm/cfdm/__init__.py
cfunits: 3.2.9 /home/sadie/cfunits/cfunits/__init__.py
cfplot: 3.0.6 /home/sadie/anaconda3/lib/python3.7/site-packages/cfplot/__init__.py
cf: 3.6.0 /home/sadie/cf-python/cf/__init__.py
Running tests from /home/sadie
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
We should remove all coupling between run directory and test module outcome.
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
調査の方向性
まず、cf/test/run_tests.py を使って、リポジトリ、そのルート、および $HOME から実行した場合の異なる結果を再現します。そのスクリプトでのテスト検出とパス処理を調べ、次に、作業ディレクトリにかかわらず同じテストスイートが選択されることを確認します。完了の条件は、起動ディレクトリが原因でテストが失敗せず、空の検出結果が誤解を招く成功を報告できないことです。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- testing-qa
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100