jerry-git / jerry-git/pytest-split

Splits invalid when collection order not deterministic

Aperta
#25 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
325
Fork
56
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

The big assumption underlying the two splitting algorithms is that the order of collected items is constant.
However, I've come across a case where this assumption was violated.
In my case I had a test parametrised with `pytest.mark.parametrize`, but the items to parametrize with would sometimes change order.

Take this example:

```python
import pytest

@pytest.mark.parametrize('name', set(['henk', 'ingrid']))
def test_hello(name):
pass
```

If you run this often enough you'll see that the order changes:

```
[2021-06-17 22:47:10] test_temp.py::test_hello[henk] PASSED [ 50%]
[2021-06-17 22:47:10] test_temp.py::test_hello[ingrid] PASSED [100%]
```

and

```
[2021-06-17 22:47:10] test_temp.py::test_hello[ingrid] PASSED [ 50%]
[2021-06-17 22:47:10] test_temp.py::test_hello[henk] PASSED [100%]
```

I'm not sure how to address this, but I think there are a few options:

1. not splitting over different values of `parametrize` for the same test. In other words, make sure that a single group will run all tests for `test_hello`.
2. try to create some deterministic order out of test cases by sorting. I'm not sure this will work in all cases tho (for example it might not work for objects)
3. do splitting on one machine, save the splits and just call `pytest` with those pre-calculated groups (so not really using this plugin as a plugin :p)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.