google / google/openhtf

Poor naming: ambiguous use of test

Open
#817 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
722
Forks
237
Avg merge
11h 31m
Merged PRs (30d)
4

Description

The opening documentation of OpenHTF does a pretty good job of differentiating the different uses of test:

* test run - A single start-to-finish execution of a specific test.
* test recipe - A test definition that may be executed multiple times, each time as a distinct test run.
* test script - A .py file that contains a test recipe.

I can get behind these definitions, in particular the test run and test recipe nomenclature. It makes a clear distinction between the static definition of a test (test recipe) and an individual instance of a test (test run).

Unfortunately, our code base doesn't do a very good job of using distinct naming that I think is a point of confusion for both test authors and framework authors/maintainers. Probably the most egrerious example is as follows:

1. We ask test authors to create a test:
```
my_test = htf.Test(
my_phase_1,
my_phase2,
etc
)
my_test.execute()
```

2. As a common practice we use the variable test as the first argument in phases:
```
@htf.measurements(...)
def my_phase_1(test):
test.measurements.my_measurement = 1 # not actually adding measurement to the test, but to the test run
test.attach(my_data, 'foo.csv') # not actually attaching to the test but test run.
test.add_output_callback(my_callback) # won't work, but not clear to a new test author why
```
Strictly speaking, this is more of a bad practice which can be improved by just discouraging use of test inside the phase.

I'd propose in a future version (or major revision) of OpenHTF to rename TestState --> TestRun and TestApi --> TestRunApi. Optionally, can also rename Test --> TestRecipe.

Somewhat related: I also am not a fan of the overall architecture with Test.execute being the main entry point. As a result, doing parallel test runs is difficult because everything is a reference of Test. i.e. Test --> TestExecutor --> TestState / PhaseExecutor, etc.

I think a better architecture would be to completely separate test execution from TestRecipe and TestRun. i.e. We dumb-down Test/TestRecipe and TestState/TestRun to just simple data objects. A test executor is responsible for taking the Test/TestRecipe and configuration data and executing a TestRun. A parallel test executor could have queue and it's job is to just pull of the queue and execute tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.