DSGN: collect tests metainfo during execution

As stated here collection of all metadata happens before tests are actually executed. This leads to performance issues that makes pytest parametrization hard to use for at least following 2 reasons:

1. Iterators (again) exactly as in already mentioned issue and in others (not only on current tracker) iterators are needed to iterate lazily otherwise pytest is not helper here.

Consider example: we want to parametrize test over array with say 10 parameters. Pytest easily deal with this creating all possible permutations from 10. But if now we need all permutations for 2 levels - we need 10 times more tests and 10 more RAM? What if we need more (in fact it is not just intentionally created test - its from real life problem; if we test function with 2-3 nested conditions and maybe nested 2-3 function calls inside we already need about 4-9 level in order to hit each situation possible). In fact even 4-5 levels (i.e. maximum is 4-5 level for nested condition possible) with 20GB RAM already not feasible (but this info from my concrete case of course).

2. Even if we managed to fit in RAM before our tests actually running we need to wait for metadata being collected. But what if we not need ALL this data? What if we add -x mark to pytest parameters? Answer is we may wait for 3-5 min for metadata being collected and right after that we may fail on the 1st test. But even if we not - one almost never need all this bunch of metainfo - only the failed ones (which count usually much lesser than all input tests count), i.e. we not even need to collect metainfo before - we even not need majority of it collected anyhow (lazily or not; at least not all of it).

And yes, I saw comment in mentioned issues about design problem, etc. Just get another ping in order to remind that without collecting metadata and treating iterators lazily pytest is not scalable.

In order to bring some thoughts here (cause redesign is always really hard and maybe here are some workarounds as well) maybe follow-ups might be somehow internally reimplemented for iterators without need to do major rewriting:
For now I am using some workaround like this:

datas = gen_data()  # data generator
@pytest.fixture(scope='module', params=len_of_datas_if_known)
def fix():
    huge_data_chunk = next(datas)
    return huge_data_chunk


@pytest.mark.parametrize('other_param', ['aaa', 'bbb'])
def test_one(fix, other_param):
    data = fix
    ...

So I'm still parametrizing fixture but over "indexes" and just in order to say pytest how many times do retest with new data. It's ugly but for now it works) Also need somehow knew count of generated data (but if we exceed this count - just will get pytests fails with StopIteration error which might be marked as passed).

Another approach is to use fixtures as factories - but here as I understand we will iterate over data inside test, so for pytest it will be as single test which is not good.

And last what I've devised here for now is to use pytest.main() in loop. Something like so:

# data_generate
# set_up test
pytest.main(['test'])

No repository files, tests, or entry points are named. Start by reading the collection and parametrization concerns in the linked issues 3070 and 4002, then define done as metadata being collected lazily or on demand while supporting iterators and early termination such as -x.