abseil / abseil/abseil-py

Test cases should be parameterizable based on command-line flags (and other things)

Open
#242 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
279
Avg merge
2d 1h
Merged PRs (30d)
1

Description

Enumeration of the cases for a parameterized test method in an `absl.testing.parameterized.TestCase` is eager: it happens at the point where the class is defined.

Under normal circumstances, this is before `googletest.main()` has been called, meaning that the sequence of cases cannot depend on the values of flags defined by `absl.flags` (or, indeed, on anything that is not meaningfully defined at the point that the `TestCase` class is created).

Thus, the following structure will not work:

```python
from absl import flags
from absl.testing import absltest
from absl.testing import parameterized

MY_FLAG = flags.DEFINE_bool(...)

def _gen_cases():
return (x for x in ...) if MY_FLAG.value else (y for y in ...) # But MY_FLAG won't have been set yet

class MyTest(parameterized.TestCase):

@parameterized.parameters(_gen_cases()):
def test_my_condition(self, case):
...

if __name__ == '__main__':
absltest.main()
```

### Possible Fix

Either of two changes could fix this: making command-line flag parsing more eager, or making test case enumeration lazier. The latter seems simpler.

Each of `parameterized.parameters()`, `parameterized.named_parameters()`, `parameterized.product()` and so on could be widened to accept, in addition to the types that it currently accepts, a `callable` that returns one of those types. If it finds its argument is callable, it could defer enumeration of the cases until some point after command-line flags have been parsed.

Then the above snippet could be amended:

```python
@parameterized.parameters(_gen_cases): # Note the generator is not called at this point
def test_my_condition(self, case):
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.