pytest-dev / pytest-dev/pytest

pytest cache files contain absolute paths

Open
#3,968 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

topic: rewrite type: enhancement
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

The .pyc files that pytest creates with the PYTEST suffix include hardcoded full paths in their embedded source representation. This means that if the file moves, it will no longer be valid if the original location is removed, and pytest will throw errors trying to import these cached files. I checked what the CPython .pyc files are doing differently (which work after moving), and it looks like this is due to how pytest writes the cache files. Here's a quick test using uncompyle6, showing the headers only (decompiled file contents are exactly the same) using a file from scipy:

uncompyle6 conftest.cpython-36.pyc
# uncompyle6 version 3.2.3
# Python bytecode 3.6 (3379)
# Decompiled from: Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
# Embedded file name: Lib\site-packages\scipy\conftest.py
# Compiled at: 2018-05-05 10:07:41
# Size of source mod 2**32: 999 bytes

uncompyle6 conftest.cpython-36-PYTEST.pyc
# uncompyle6 version 3.2.3
# Python bytecode 3.6 (3379)p
# Decompiled from: Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
# Embedded file name: C:\apps\miniconda_x\envs\up\lib\site-packages\scipy\conftest.py
# Compiled at: 2018-05-05 10:07:41
# Size of source mod 2**32: 999 bytes

You can see from the "Embedded file name" line that a CPython .pyc contains a relative path, but the PYTEST version contains an absolute path. Systems such as conda typically distribute compiled components, and it would be nice to be able to distribute PYTEST files this way, and more generally be able to move environments.

Here's an oversimplified fix:

import sys
from pathlib import Path

relative_path = Path(fn.strpath).relative_to(sys.prefix)

and use this value for fn.strpath in https://github.com/pytest-dev/pytest/blob/e0539e6ede6b9bcfcb7ff9c39602ea6ff5c21537/src/_pytest/assertion/rewrite.py#L414.

I can make a PR, but it wasn't clear from my reading if the full path is explicitly required somehow by pytest, and figured I'd ask as an issue first.

Environment:

x64 Windows 10, pytest 3.7.4, Python 3.6.6, conda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/_pytest/assertion/rewrite.py around line 414, where the issue identifies pytest's use of fn.strpath when writing rewritten bytecode. Compare the generated PYTEST cache metadata with CPython's relative embedded path behavior. Done means moved pytest cache files remain importable after the original absolute location is removed, without breaking pytest's path handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.