jerry-git / jerry-git/pytest-split

Splits invalid when collection order not deterministic

オープン
#25 コメント 6 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
325
フォーク
56
PR マージ指標
30日以内にマージされた PR はありません

説明

The big assumption underlying the two splitting algorithms is that the order of collected items is constant.
However, I've come across a case where this assumption was violated.
In my case I had a test parametrised with `pytest.mark.parametrize`, but the items to parametrize with would sometimes change order.

Take this example:

```python
import pytest

@pytest.mark.parametrize('name', set(['henk', 'ingrid']))
def test_hello(name):
pass
```

If you run this often enough you'll see that the order changes:

```
[2021-06-17 22:47:10] test_temp.py::test_hello[henk] PASSED [ 50%]
[2021-06-17 22:47:10] test_temp.py::test_hello[ingrid] PASSED [100%]
```

and

```
[2021-06-17 22:47:10] test_temp.py::test_hello[ingrid] PASSED [ 50%]
[2021-06-17 22:47:10] test_temp.py::test_hello[henk] PASSED [100%]
```

I'm not sure how to address this, but I think there are a few options:

1. not splitting over different values of `parametrize` for the same test. In other words, make sure that a single group will run all tests for `test_hello`.
2. try to create some deterministic order out of test cases by sorting. I'm not sure this will work in all cases tho (for example it might not work for objects)
3. do splitting on one machine, save the splits and just call `pytest` with those pre-calculated groups (so not really using this plugin as a plugin :p)

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Reproduce the order variation with the pytest.mark.parametrize example, then trace the two splitting algorithms; no file paths or tests are provided. Compare the proposed handling strategies, and consider the issue done when repeated runs assign collected tests to consistent groups without breaking pytest collection.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
testing
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
説明が足りない
初心者へのやさしさ
38/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。