pytest-dev / pytest-dev/pytest
pytest cache files contain absolute paths
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
The .pyc files that pytest creates with the PYTEST suffix include hardcoded full paths in their embedded source representation. This means that if the file moves, it will no longer be valid if the original location is removed, and pytest will throw errors trying to import these cached files. I checked what the CPython .pyc files are doing differently (which work after moving), and it looks like this is due to how pytest writes the cache files. Here's a quick test using uncompyle6, showing the headers only (decompiled file contents are exactly the same) using a file from scipy:
uncompyle6 conftest.cpython-36.pyc
# uncompyle6 version 3.2.3
# Python bytecode 3.6 (3379)
# Decompiled from: Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
# Embedded file name: Lib\site-packages\scipy\conftest.py
# Compiled at: 2018-05-05 10:07:41
# Size of source mod 2**32: 999 bytes
uncompyle6 conftest.cpython-36-PYTEST.pyc
# uncompyle6 version 3.2.3
# Python bytecode 3.6 (3379)p
# Decompiled from: Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
# Embedded file name: C:\apps\miniconda_x\envs\up\lib\site-packages\scipy\conftest.py
# Compiled at: 2018-05-05 10:07:41
# Size of source mod 2**32: 999 bytes
You can see from the "Embedded file name" line that a CPython .pyc contains a relative path, but the PYTEST version contains an absolute path. Systems such as conda typically distribute compiled components, and it would be nice to be able to distribute PYTEST files this way, and more generally be able to move environments.
Here's an oversimplified fix:
import sys
from pathlib import Path
relative_path = Path(fn.strpath).relative_to(sys.prefix)
and use this value for fn.strpath in https://github.com/pytest-dev/pytest/blob/e0539e6ede6b9bcfcb7ff9c39602ea6ff5c21537/src/_pytest/assertion/rewrite.py#L414.
I can make a PR, but it wasn't clear from my reading if the full path is explicitly required somehow by pytest, and figured I'd ask as an issue first.
Environment:
x64 Windows 10, pytest 3.7.4, Python 3.6.6, conda
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/_pytest/assertion/rewrite.py around line 414, where the issue identifies pytest's use of fn.strpath when writing rewritten bytecode. Compare the generated PYTEST cache metadata with CPython's relative embedded path behavior. Done means moved pytest cache files remain importable after the original absolute location is removed, without breaking pytest's path handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100