pytest-dev / pytest-dev/pytest
Fixture discovery scans every attribute of every registered plugin, on every Config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
generated by opus - will make a better description after i conclude the experiment where this was noted
Summary
Registering a plugin makes FixtureManager.parsefactories() walk dir(plugin) and run _check_for_wrapped_fixture() on every attribute, looking for fixtures. Most plugins have none — reporters, loggers, managers, the collection machinery — so the whole scan is pure overhead, and it is repeated for every Config that gets built.
In a default session, 22 of 32 registered plugins define no fixtures, and 877 attributes are read and validated for nothing.
This is cheap enough to ignore for a single run. It is not cheap when something builds many configs: pytest's own test suite does exactly that, thousands of times, through pytester.
Measurements
Profiling one nested Config build + tiny run (30 iterations, cProfile, cumulative):
pluginmanager.register 0.790s of 1.098s total (72%)
└ fixtures.parsefactories 0.433s (39%)
└ _check_for_wrapped_fixture 39,690 calls (~1,320 per config)
└ inspect.signature 4,830 calls
Attribute introspections per config build:
| scans | |
|---|---|
| default plugin set | 1144 |
…plus the terminal plugin |
1323 |
Registering TerminalReporter alone costs 179 attribute reads on an object that has never held a fixture — dir() on the class walks the whole MRO, and each entry goes through _find_wrapped_fixture_def().
Impact
I hacked in an opt-out to see what it was worth: a plugin sets __pytest_no_fixtures__ = True and parsefactories returns immediately. Applied to the 20 core plugin modules that define no fixtures, plus TerminalReporter and the config objects:
- attribute introspections 1144 → 589, and 1323 → 589 with the terminal loaded (the reporter's scan disappears entirely)
- per-config cost 10.33ms → 8.53ms, and 12.76ms → 10.35ms with the terminal
A/B on identical trees, one commit apart, running pytest's own suite:
testing/ complete (-n 8) |
|
|---|---|
| without opt-out | 63.54s |
| with opt-out | 50.50s |
~20% off pytest's own test suite, with no test changes at all. The saving is entirely in pytester-driven tests, each of which builds a config and registers every plugin.
To be clear about who benefits: an ordinary single-session run builds one Config, so this is worth ~2ms and nobody would notice. It matters for anything that builds configs repeatedly — pytester, and therefore pytest's own CI, most of all.
The flag is a hack
I am not proposing __pytest_no_fixtures__ as the fix. It is the crudest thing that could demonstrate the cost, and it has an obvious failure mode: add a fixture to a plugin that has opted out, and it is silently never collected. I guarded that locally with a test that walks every plugin carrying the flag and fails if any of them actually defines a fixture — but needing such a guard is itself the argument against the design.
If this is worth extracting, better directions than a manual flag:
- Make discovery push-based.
@pytest.fixtureknows the function it decorates at decoration time; recording into a per-module registry there would make discoveryO(fixtures)instead ofO(attributes), and delete the scan rather than skip it. - Cache per holder object. Plugin modules are the same objects across every
Configin a process, and their fixture sets cannot change between configs, so the discovery result is memoizable — this needs no per-plugin declaration and cannot go stale the way a flag can. - Scan
vars()rather thandir()where the holder is a module, avoiding the MRO walk and inherited attributes entirely. - Invert it — have plugins opt in to being scanned — which has the same silent-drop hazard as the flag and is probably no better.
(2) looks like the best cost/benefit: no API surface, no way to get it wrong, and it removes the repeat cost that actually hurts.
Context
Found while profiling nested Config construction for an experimental in-process test-running API (#14809), where the per-config cost is paid on every single run rather than once per session. The finding is not specific to that work — the same scan runs in every pytest session and every pytester run.
Happy to open a PR for whichever direction maintainers prefer; I would rather not land the flag version.
Per our AI contribution policy: this was investigated with Claude Code under my direction and review.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with FixtureManager.parsefactories(), _check_for_wrapped_fixture(), and pluginmanager.register while profiling repeated Config construction through pytester. Compare the proposed discovery approaches using the measurements in the issue, then run the testing/ suite and verify that fixture discovery remains correct while repeated Config builds avoid unnecessary scans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100