pytest-dev / pytest-dev/pytest
pytest crashing with ptest-xdist
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
When running pytest in parallel w/ pytest-xdist, I'm seeing the follow stack trace:
/opt/pyenv/versions/3.10.12/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:688: in _load_unlocked
???
/usr/local/lib/gravity/poetry/venvs/MXt4Tns3-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:174: in exec_module
co = _read_pyc(fn, pyc, state.trace)
/usr/local/lib/gravity/poetry/venvs/MXt4Tns3-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:374: in _read_pyc
with fp:
E OSError: [Errno 5] Input/output error
Looking at the code, it seems like we're missing some exception handling.
When I put the entire context in the try/except, everything works as expected.
Seems like this is a best attempt, as if the open fails, we return None.
def _read_pyc(
source: Path, pyc: Path, trace: Callable[[str], None] = lambda x: None
) -> types.CodeType | None:
"""Possibly read a pytest pyc containing rewritten code.
Return rewritten code if successful or None if not.
"""
try:
fp = open(pyc, "rb")
except OSError:
return None
with fp: <-------------------- shouldn't this be in the try/except?
try:
stat_result = os.stat(source)
mtime = int(stat_result.st_mtime)
size = stat_result.st_size
data = fp.read(16)
except OSError as e:
trace(f"_read_pyc({source}): OSError {e}")
return None
# Check for invalid or out of date pyc file.
if len(data) != (16):
trace(f"_read_pyc({source}): invalid pyc (too short)")
return None
if data[:4] != importlib.util.MAGIC_NUMBER:
trace(f"_read_pyc({source}): invalid pyc (bad magic number)")
return None
if data[4:8] != b"\x00\x00\x00\x00":
trace(f"_read_pyc({source}): invalid pyc (unsupported flags)")
return None
mtime_data = data[8:12]
if int.from_bytes(mtime_data, "little") != mtime & 0xFFFFFFFF:
trace(f"_read_pyc({source}): out of date")
return None
size_data = data[12:16]
if int.from_bytes(size_data, "little") != size & 0xFFFFFFFF:
trace(f"_read_pyc({source}): invalid pyc (incorrect size)")
return None
try:
co = marshal.load(fp)
except Exception as e:
trace(f"_read_pyc({source}): marshal.load error {e}")
return None
if not isinstance(co, types.CodeType):
trace(f"_read_pyc({source}): not a code object")
return None
return co
installed python modules:
dev@gvt-dev:~/repo/ws/global-config-manager$ poetry run pip list
Package Version
------------------------- -------------
aniso8601 10.0.1
astroid 2.15.8
async-timeout 5.0.1
atlassian-python-api 4.0.7
attrs 25.4.0
backports.asyncio.runner 1.2.0
backports.zstd 1.3.0
beautifulsoup4 4.14.2
blinker 1.9.0
brotli 1.2.0
certifi 2025.11.12
cffi 2.0.0
cfgv 3.4.0
charset-normalizer 3.4.4
circus 0.19.0
click 8.3.0
confluent-kafka 2.8.0
coverage 7.11.3
cramjam 2.11.0
crypto 1.4.1
cryptography 45.0.7
deepdiff 8.6.1
Deprecated 1.3.1
dill 0.4.0
distlib 0.4.0
enum34 1.1.10
exceptiongroup 1.3.0
execnet 2.1.2
fakeredis 2.32.1
filelock 3.20.0
flake8 7.3.0
flake8-tidy-imports 4.12.0
Flask 3.1.2
Flask-Compress 1.23
flask-restx 1.3.2
gevent 25.9.1
greenlet 3.2.4
grpcio 1.76.0
grpcio-tools 1.71.2
hvac 2.3.0
icdiff 2.0.7
identify 2.6.15
idna 3.11
importlib_resources 6.5.2
iniconfig 2.3.0
isort 5.13.2
itsdangerous 2.2.0
Jinja2 3.1.6
jmespath 1.0.1
jsonpickle 4.1.1
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
lazy-object-proxy 1.12.0
libyang 2.99999.99999
lxml 6.0.2
MarkupSafe 3.0.3
mccabe 0.7.0
mmh3 4.1.0
Naked 0.1.32
netaddr 1.3.0
newrelic 11.2.0
nodeenv 1.9.1
oauthlib 3.3.1
orderly-set 5.5.0
packaging 25.0
pip 25.1.1
platformdirs 4.5.0
pluggy 1.6.0
pprintpp 0.4.0
pre_commit 4.4.0
protobuf 5.29.5
psutil 7.1.3
psycopg2-binary 2.9.11
py 1.11.0
pyang 2.7.1
pyangbind 0.8.7
pycodestyle 2.14.0
pycparser 2.23
pycryptodome 3.23.0
pyflakes 3.4.0
Pygments 2.19.2
pylint 2.17.7
pylint-exit 1.2.0
pyOpenSSL 25.2.0
pyroaring 1.0.3
pytest 8.4.2
pytest-asyncio 1.3.0
pytest-cov 6.3.0
pytest-flask 1.3.0
pytest-forked 1.6.0
pytest-icdiff 0.9
pytest-mock 3.15.1
pytest-sugar 1.1.1
pytest-xdist 3.8.0
python-consul 1.1.0
python-generate-mac 1.3.1
python-snappy 0.7.3
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
redis 5.2.1
referencing 0.37.0
regex 2025.11.3
requests 2.32.5
requests-mock 1.12.1
requests-oauthlib 2.0.0
retrying 1.4.2
rpds-py 0.28.0
setuptools 80.9.0
shellescape 3.8.1
simplekv 0.14.1
six 1.17.0
sortedcontainers 2.4.0
soupsieve 2.8
SQLAlchemy 1.4.54
sqlalchemy-cockroachdb 1.4.4
SQLAlchemy-Utils 0.38.3
termcolor 3.2.0
tomli 2.3.0
tomlkit 0.13.3
tornado 6.5.2
typing_extensions 4.15.0
urllib3 2.5.0
uWSGI 2.0.31
virtualenv 20.35.4
Werkzeug 3.1.4
wrapt 1.17.3
zope.event 6.1
zope.interface 8.1
running on mac os, in docker
$ uname -a
Linux gvt-dev 6.10.14-linuxkit #1 SMP Wed Sep 10 06:47:45 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading _pytest/assertion/rewrite.py and the _read_pyc entry point shown in the traceback. Reproduce pytest running in parallel with pytest-xdist, then verify that an OSError during the pyc context handling no longer crashes the run and that the existing fallback behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100