pyodide / pyodide/pytest-pyodide

pytest_pyodide.hook make collection very slow even if @run_in_pyodide is never used

Open
#178 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
38
Forks
20
Avg merge
1d 23h
Merged PRs (30d)
1

Description

TL;DR: if you have a moderately-sized test suite, hook.py:pytest_pycollect_makemodule make collection noticeably slower.

I bumbed into issue when working on SPy.
SPy has 65 test files and ~5k total test lines:

❯ cloc --vcs=git spy/tests
      65 text files.
      65 unique files.                              
       3 files ignored.

github.com/AlDanial/cloc v 1.98  T=0.13 s (509.0 files/s, 95704.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          65           1813           4653           5756

I have pytest-pyodide installed in my venv, and collection takes ~3s:

❯ pytest -q --collect-only
[...]
1203 tests collected in 3.2s

If I manually comment out this function, I shave more that 1 second:
https://github.com/pyodide/pytest-pyodide/blob/f97d345d4410c594d8f3748c909d3460a531a9ba/pytest_pyodide/hook.py#L243-L251

❯ pytest -q --collect-only
[...]
1203 tests collected in 1.29s

I don't fully understand what pytest_collect_makemodule does, but it seems related to @run_in_pyodide.
Note that my tests use this functionality only in one test file:

❯ rg 'run_in_pyodide' spy/tests
spy/tests/test_llwasm.py
2:from pytest_pyodide import run_in_pyodide  # type: ignore
49:            self.run_in_pyodide_maybe = run_in_pyodide
53:            self.run_in_pyodide_maybe = lambda fn: fn
63:        @self.run_in_pyodide_maybe
85:        @self.run_in_pyodide_maybe
105:        @self.run_in_pyodide_maybe
124:        @self.run_in_pyodide_maybe
148:        @self.run_in_pyodide_maybe
173:        @self.run_in_pyodide_maybe
207:        @self.run_in_pyodide_maybe

I tried to do this hack and it seems to help a lot, although again I'm not really sure to understand what it does:

def pytest_pycollect_makemodule(module_path: Path, parent: Collector) -> None:
    source = module_path.read_bytes()
    if b"run_in_pyodide" in source:
        strfn = str(module_path)
        tree = ast.parse(source, filename=strfn)
        ORIGINAL_MODULE_ASTS[strfn] = tree
        tree2 = deepcopy(tree)
        rewrite_asserts(tree2, source, strfn, REWRITE_CONFIG)
        REWRITTEN_MODULE_ASTS[strfn] = tree2
    orig_pytest_pycollect_makemodule(module_path, parent)
❯ pytest -q --collect-only
[...]
1203 tests collected in 1.34s

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in hook.py at pytest_pycollect_makemodule and reproduce the report with pytest -q --collect-only. Compare collection with and without run_in_pyodide usage, using the reported SPy test layout and the linked function as context. Done means collection no longer pays the current overhead for modules that do not use run_in_pyodide while preserving the existing behavior for modules that do.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.