Poor naming: ambiguous use of test
- Dominant language
- Python
- Stars
- 722
- Forks
- 237
- Avg merge
- 11h 31m
- Merged PRs (30d)
- 4
Description
The opening documentation of OpenHTF does a pretty good job of differentiating the different uses of test:
* test run - A single start-to-finish execution of a specific test.
* test recipe - A test definition that may be executed multiple times, each time as a distinct test run.
* test script - A .py file that contains a test recipe.
I can get behind these definitions, in particular the test run and test recipe nomenclature. It makes a clear distinction between the static definition of a test (test recipe) and an individual instance of a test (test run).
Unfortunately, our code base doesn't do a very good job of using distinct naming that I think is a point of confusion for both test authors and framework authors/maintainers. Probably the most egrerious example is as follows:
1. We ask test authors to create a test:
```
my_test = htf.Test(
my_phase_1,
my_phase2,
etc
)
my_test.execute()
```
2. As a common practice we use the variable test as the first argument in phases:
```
@htf.measurements(...)
def my_phase_1(test):
test.measurements.my_measurement = 1 # not actually adding measurement to the test, but to the test run
test.attach(my_data, 'foo.csv') # not actually attaching to the test but test run.
test.add_output_callback(my_callback) # won't work, but not clear to a new test author why
```
Strictly speaking, this is more of a bad practice which can be improved by just discouraging use of test inside the phase.
I'd propose in a future version (or major revision) of OpenHTF to rename TestState --> TestRun and TestApi --> TestRunApi. Optionally, can also rename Test --> TestRecipe.
Somewhat related: I also am not a fan of the overall architecture with Test.execute being the main entry point. As a result, doing parallel test runs is difficult because everything is a reference of Test. i.e. Test --> TestExecutor --> TestState / PhaseExecutor, etc.
I think a better architecture would be to completely separate test execution from TestRecipe and TestRun. i.e. We dumb-down Test/TestRecipe and TestState/TestRun to just simple data objects. A test executor is responsible for taking the Test/TestRecipe and configuration data and executing a TestRun. A parallel test executor could have queue and it's job is to just pull of the queue and execute tests.
Contributor guide
Assessment
This issue has not been assessed yet.