Test cases should be parameterizable based on command-line flags (and other things)
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 279
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 1
Description
Enumeration of the cases for a parameterized test method in an `absl.testing.parameterized.TestCase` is eager: it happens at the point where the class is defined.
Under normal circumstances, this is before `googletest.main()` has been called, meaning that the sequence of cases cannot depend on the values of flags defined by `absl.flags` (or, indeed, on anything that is not meaningfully defined at the point that the `TestCase` class is created).
Thus, the following structure will not work:
```python
from absl import flags
from absl.testing import absltest
from absl.testing import parameterized
MY_FLAG = flags.DEFINE_bool(...)
def _gen_cases():
return (x for x in ...) if MY_FLAG.value else (y for y in ...) # But MY_FLAG won't have been set yet
class MyTest(parameterized.TestCase):
@parameterized.parameters(_gen_cases()):
def test_my_condition(self, case):
...
if __name__ == '__main__':
absltest.main()
```
### Possible Fix
Either of two changes could fix this: making command-line flag parsing more eager, or making test case enumeration lazier. The latter seems simpler.
Each of `parameterized.parameters()`, `parameterized.named_parameters()`, `parameterized.product()` and so on could be widened to accept, in addition to the types that it currently accepts, a `callable` that returns one of those types. If it finds its argument is callable, it could defer enumeration of the cases until some point after command-line flags have been parsed.
Then the above snippet could be amended:
```python
@parameterized.parameters(_gen_cases): # Note the generator is not called at this point
def test_my_condition(self, case):
```
Contributor guide
Assessment
This issue has not been assessed yet.