pytest-dev / pytest-dev/pytest

Terminal output fails with faulthandler + eventlet

Open
#5,725 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

platform: mac topic: reporting type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

There seems to be an issue with Pytest 5 when testing codebases using eventlet. It only happens when faulthandler is enabled (now the default).

I first saw the issue when running tests against my Nameko service, and it prevented me from seeing my test failures, because the output was truncated.

I narrowed down the reproduction case to just running pytest --help. The effect is to truncate the terminal output, with the following error message sometimes interspersed near the end of wherever the output got up to:

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BlockingIOError: [Errno 35] write could not complete without blocking

this is clearly a race condition, as this error can appear in random parts of the output, and only some of the time. However I did manage to force a similar error to be displayed, via pytest --help | cat which ended with:

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

To reproduce:

pip install 'pytest>=5.0' eventlet
conftest.py:

def pytest_configure():
    # make sure we monkey_patch before `--help` is run
    import eventlet
    eventlet.monkey_patch()

Random truncation of output:

pytest --help

some of the time this includes the BlockingIOError above.

To force the error:

pytest --help | cat
usage: pytest [options] [file_or_dir] [file_or_dir] [...]

positional arguments:
  file_or_dir

general:
  -k EXPRESSION         only run tests which match the given substring
...
                        'extra_keyword_matcTraceback (most recent call last):
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/bin/pytest", line 10, in <module>
    sys.exit(main())
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/_pytest/config/__init__.py", line 74, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/_pytest/helpconfig.py", line 134, in pytest_cmdline_main
    showhelp(config)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/_pytest/helpconfig.py", line 179, in showhelp
    tw.line(wrapped[0])
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/py/_io/terminalwriter.py", line 273, in line
    self.write('\n')
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/py/_io/terminalwriter.py", line 256, in write
    write_out(self._file, markupmsg)
  File "/Users/me/.virtualenvs/eventlet-pytest-bug/lib/python3.7/site-packages/py/_io/terminalwriter.py", line 421, in write_out
    fil.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

There's no error without faulthandler enabled:

pytest -p no:faulthandler --help | cat
<entire help message shown>

No error before pytest 5:

pip install 'pytest<=5'
pytest --help | cat
<entire help message shown>

I can only reproduce this on OSX. Could not reproduce on Ubuntu. Seems to be independent of Python version and eventlet version. I git bisected the first pytest commit to have the issue, and found https://github.com/pytest-dev/pytest/commit/a37b902afea216216 where pytest-faulthandler was integrated into pytest. But I could not find an automated method to detect the failure, as I needed the real terminal for the issue to show itself.

My setup:
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce on macOS with the shown conftest.py, eventlet monkey-patching, and pytest --help | cat, comparing pytest 5 with -p no:faulthandler. Start at _pytest/helpconfig.py and py/_io/terminalwriter.py, then inspect the faulthandler integration commit. Done means help output is no longer randomly truncated or accompanied by BlockingIOError/BrokenPipeError in this setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.