pyodide / pyodide/pytest-pyodide
pytest_pyodide.hook make collection very slow even if @run_in_pyodide is never used
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38
- Forks
- 20
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 1
Description
TL;DR: if you have a moderately-sized test suite, hook.py:pytest_pycollect_makemodule make collection noticeably slower.
I bumbed into issue when working on SPy.
SPy has 65 test files and ~5k total test lines:
❯ cloc --vcs=git spy/tests
65 text files.
65 unique files.
3 files ignored.
github.com/AlDanial/cloc v 1.98 T=0.13 s (509.0 files/s, 95704.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Python 65 1813 4653 5756
I have pytest-pyodide installed in my venv, and collection takes ~3s:
❯ pytest -q --collect-only
[...]
1203 tests collected in 3.2s
If I manually comment out this function, I shave more that 1 second:
https://github.com/pyodide/pytest-pyodide/blob/f97d345d4410c594d8f3748c909d3460a531a9ba/pytest_pyodide/hook.py#L243-L251
❯ pytest -q --collect-only
[...]
1203 tests collected in 1.29s
I don't fully understand what pytest_collect_makemodule does, but it seems related to @run_in_pyodide.
Note that my tests use this functionality only in one test file:
❯ rg 'run_in_pyodide' spy/tests
spy/tests/test_llwasm.py
2:from pytest_pyodide import run_in_pyodide # type: ignore
49: self.run_in_pyodide_maybe = run_in_pyodide
53: self.run_in_pyodide_maybe = lambda fn: fn
63: @self.run_in_pyodide_maybe
85: @self.run_in_pyodide_maybe
105: @self.run_in_pyodide_maybe
124: @self.run_in_pyodide_maybe
148: @self.run_in_pyodide_maybe
173: @self.run_in_pyodide_maybe
207: @self.run_in_pyodide_maybe
I tried to do this hack and it seems to help a lot, although again I'm not really sure to understand what it does:
def pytest_pycollect_makemodule(module_path: Path, parent: Collector) -> None:
source = module_path.read_bytes()
if b"run_in_pyodide" in source:
strfn = str(module_path)
tree = ast.parse(source, filename=strfn)
ORIGINAL_MODULE_ASTS[strfn] = tree
tree2 = deepcopy(tree)
rewrite_asserts(tree2, source, strfn, REWRITE_CONFIG)
REWRITTEN_MODULE_ASTS[strfn] = tree2
orig_pytest_pycollect_makemodule(module_path, parent)
❯ pytest -q --collect-only
[...]
1203 tests collected in 1.34s
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in hook.py at pytest_pycollect_makemodule and reproduce the report with pytest -q --collect-only. Compare collection with and without run_in_pyodide usage, using the reported SPy test layout and the linked function as context. Done means collection no longer pays the current overhead for modules that do not use run_in_pyodide while preserving the existing behavior for modules that do.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100