pytest-dev / pytest-dev/pytest

Pytest 7.4.0 causing skip reason for long test names to disappear in small terminal

Open
#11,146 9 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: regression
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hi, after updating to pytest-7.4.0, I noticed that skip reasons for long tests names either get cutoff or disappear when running pytest in a small width terminal or piping output to a file.

I'm pretty sure this is due to https://github.com/pytest-dev/pytest/pull/10958/files. Since the test name is longer than the terminal width, the extra whitespace that is added to pad the line gets wrapped and then dropped, causing the removal later to be wrong.

tinyrepro.py:

import pytest

@pytest.mark.skip("Random skip reason")
def test_really_really_really_really_really_really_really_long_test_name():
    assert 1 == 1


@pytest.mark.skip("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.")
def test_really_really_really_really_really_really_really_long_test_name2():
    assert 1 == 1

Output from pytest tinyrepro.py -vv when my terminal has width 236

=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.11.3, pytest-7.4.0, pluggy-1.2.0 -- /Users/csl/opt/anaconda3/envs/delete/bin/python
cachedir: .pytest_cache
rootdir: /Users/csl/zzzzzzzz/random-testing
collected 2 items                                                                                                                                                                                                                          

tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name SKIPPED [0.0001s] (Random skip reason)                                                                                                            [ 50%]
tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name2 SKIPPED [0.0001s] (Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.)                                                                                                                                     [100%]

============================================================================================================ 2 skipped in 0.01s ============================================================================================================

Output with terminal width 90:

================================== test session starts ===================================
platform darwin -- Python 3.11.3, pytest-7.4.0, pluggy-1.2.0 -- /Users/csl/opt/anaconda3/envs/delete/bin/python
cachedir: .pytest_cache
rootdir: /Users/csl/zzzzzzzz/random-testing
collected 2 items                                                                        

tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name SKIPPED [0.0002s] [ 50%]
tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name2 SKIPPED [0.0001s]ncididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum
dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum.)                    [100%]

=================================== 2 skipped in 0.00s ===================================

Contents of output.txt from pytest tinyrepro.py -vv > output.txt

============================= test session starts ==============================
platform darwin -- Python 3.11.3, pytest-7.4.0, pluggy-1.2.0 -- /Users/csl/opt/anaconda3/envs/delete/bin/python
cachedir: .pytest_cache
rootdir: /Users/csl/zzzzzzzz/random-testing
collecting ... collected 2 items

tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name SKIPPED [0.0001s] [ 50%]
tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name2 SKIPPED [0.0001s]d tempor incididunt ut labore et dolore
magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco
laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor
in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa
qui officia deserunt mollit anim id est laborum.)                        [100%]

============================== 2 skipped in 0.00s ==============================

With pytest-7.3.2, terminal width 90, pytest tinyrepro.py -vv looks like

================================== test session starts ===================================
platform darwin -- Python 3.11.3, pytest-7.3.2, pluggy-1.2.0 -- /Users/csl/opt/anaconda3/envs/delete/bin/python
cachedir: .pytest_cache
rootdir: /Users/csl/zzzzzzzz/random-testing
collected 2 items                                                                        

tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name SKIPPED [0.0002s] (Random skip reason) [ 50%]
tinyrepro.py::test_really_really_really_really_really_really_really_long_test_name2 SKIPPED [0.0001s] (Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.) [100%]

=================================== 2 skipped in 0.01s ===================================

Output of pip list:

sh: /opt/homebrew/bin/pip: bad interpreter: /opt/homebrew/opt/python35/bin/python3.5: no such file or directory
Package    Version
---------- -------
iniconfig  2.0.0
packaging  23.1
pip        23.1.2
pluggy     1.2.0
pytest     7.4.0
setuptools 67.8.0
wheel      0.38.4

Pytest version: 7.4.0
OS: macOS Ventura 13.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with tinyrepro.py using a narrow terminal width and redirected output, then inspect the behavior introduced by pull request 10958. Add regression coverage for long test names and skip reasons, and verify that the complete skip reason remains visible in both terminal and piped output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.