pytest-dev / pytest-dev/pytest-xdist

pytest parallel process failing with INTERNALERROR in azure machine

Open
#1,194 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

pytest parallel processes are failing with INTERNALERROR

Verisons using:
pytest==8.3.1
pytest-xdist==3.6.1
azure-cli==2.70.0
Python (Linux) 3.9.21

Command ran:
pytest
-v
-n 4
-m 'sit'
--cache-clear
--json-report
......
......

But the same command with-n 1 works fine

INTERNALERROR> def worker_internal_error(
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR> E                 result = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 174, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 195, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR> E                 res = yield
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E                 teardown.throw(outcome._exception)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/faulthandler.py", line 87, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 113, in pytest_runtest_protocol
INTERNALERROR> E                 runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line [132](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:133), in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "call", log))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/_pytest/runner.py", line 246, in call_and_report
INTERNALERROR> E                 ihook.pytest_runtest_logreport(report=report)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 255, in pytest_runtest_logreport
INTERNALERROR> E                 self.sendevent("testreport", data=data)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/xdist/remote.py", line 98, in sendevent
INTERNALERROR> E                 self.channel.send((name, kwargs))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 912, in send
INTERNALERROR> E                 self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1190, in _send
INTERNALERROR> E                 message.to_io(self._io)
INTERNALERROR> E               File "/home/runner/.local/lib/python3.9/site-packages/execnet/gateway_base.py", line 576, in to_io
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/.local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.5.64.33'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
INTERNALERROR> E                 header = struct.pack("!bii", self.msgcode, self.channelid, len(self.data))
INTERNALERROR> E             struct.error: 'i' format requires -2147483648 <= number <= 2147483647
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py:232: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR>     teardown.throw(outcome._exception)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line [138](https://github.com/yugabyte/yugabyte-cloud/actions/runs/14407462005/job/40409253750#step:20:139), in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/runner/.local/lib/python3.9/site-packages/xdist/dsession.py", line 217, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported pytest command with -n 1 and -n 4, then read the xdist paths shown in xdist/remote.py and xdist/dsession.py around worker reporting and channel sends. Investigate the execnet gateway_base.py struct.pack failure; done means the parallel command completes without INTERNALERROR while preserving test reports.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.