pytest-dev / pytest-dev/pytest

Remove parametrize decorators (and perhaps docstrings) from tracebacks

Open
#8,484 3 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: parametrize topic: reporting topic: tracebacks
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Consider a test like this:

@pytest.mark.parametrize('text1, text2, equal', [
    # schemes
    ("http://en.google.com/blah/*/foo",
     "https://en.google.com/blah/*/foo",
     False),
    ("https://en.google.com/blah/*/foo",
     "https://en.google.com/blah/*/foo",
     True),
    ("https://en.google.com/blah/*/foo",
     "ftp://en.google.com/blah/*/foo",
     False),

    # subdomains
    ("https://en.google.com/blah/*/foo",
     "https://fr.google.com/blah/*/foo",
     False),
    ("https://www.google.com/blah/*/foo",
     "https://*.google.com/blah/*/foo",
     False),
    ("https://*.google.com/blah/*/foo",
     "https://*.google.com/blah/*/foo",
     True),

    # domains
    ("http://en.example.com/blah/*/foo",
     "http://en.google.com/blah/*/foo",
     False),

    # ports
    ("http://en.google.com:8000/blah/*/foo",
     "http://en.google.com/blah/*/foo",
     False),
    ("http://fr.google.com:8000/blah/*/foo",
     "http://fr.google.com:8000/blah/*/foo",
     True),
    ("http://en.google.com:8000/blah/*/foo",
     "http://en.google.com:8080/blah/*/foo",
     False),

    # paths
    ("http://en.google.com/blah/*/foo",
     "http://en.google.com/blah/*",
     False),
    ("http://en.google.com/*",
     "http://en.google.com/",
     False),
    ("http://en.google.com/*",
     "http://en.google.comm/*",
     True),

    # all_urls
    ("<all_urls>",
     "<all_urls>",
     True),
    ("<all_urls>",
     "http://*/*",
     False)
])
def test_equal(text1, text2, equal):
    pat1 = urlmatch.UrlPattern(text1)
    pat2 = urlmatch.UrlPattern(text2)

    assert (pat1 == pat2) == equal
    assert (hash(pat1) == hash(pat2)) == equal

If a parametrized value fails, we get:

_______________________________________________________________________________________ test_equal[http://en.google.com/*-http://en.google.comm/*-True] _______________________________________________________________________________________

text1 = 'http://en.google.com/*', text2 = 'http://en.google.comm/*', equal = True

    @pytest.mark.parametrize('text1, text2, equal', [
        # schemes
        ("http://en.google.com/blah/*/foo",
         "https://en.google.com/blah/*/foo",
         False),
        ("https://en.google.com/blah/*/foo",
         "https://en.google.com/blah/*/foo",
         True),
        ("https://en.google.com/blah/*/foo",
         "ftp://en.google.com/blah/*/foo",
         False),
    
        # subdomains
        ("https://en.google.com/blah/*/foo",
         "https://fr.google.com/blah/*/foo",
         False),
        ("https://www.google.com/blah/*/foo",
         "https://*.google.com/blah/*/foo",
         False),
        ("https://*.google.com/blah/*/foo",
         "https://*.google.com/blah/*/foo",
         True),
    
        # domains
        ("http://en.example.com/blah/*/foo",
         "http://en.google.com/blah/*/foo",
         False),
    
        # ports
        ("http://en.google.com:8000/blah/*/foo",
         "http://en.google.com/blah/*/foo",
         False),
        ("http://fr.google.com:8000/blah/*/foo",
         "http://fr.google.com:8000/blah/*/foo",
         True),
        ("http://en.google.com:8000/blah/*/foo",
         "http://en.google.com:8080/blah/*/foo",
         False),
    
        # paths
        ("http://en.google.com/blah/*/foo",
         "http://en.google.com/blah/*",
         False),
        ("http://en.google.com/*",
         "http://en.google.com/",
         False),
        ("http://en.google.com/*",
         "http://en.google.comm/*",
         True),
    
        # all_urls
        ("<all_urls>",
         "<all_urls>",
         True),
        ("<all_urls>",
         "http://*/*",
         False)
    ])
    def test_equal(text1, text2, equal):
        pat1 = urlmatch.UrlPattern(text1)
        pat2 = urlmatch.UrlPattern(text2)
    
>       assert (pat1 == pat2) == equal
E       AssertionError: assert (equals failed
E         qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.com/*')   qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.comm/*') ) == True

tests/unit/utils/test_urlmatch.py:690: AssertionError

However, the whole parametrize decorator really is irrelevant - it makes the output very long, contains a lot of values which actually aren't the one the test did run with, and hides the very relevant text1 = 'http://en.google.com/*', text2 = 'http://en.google.comm/*', equal = True at the top.

Ideally, we'd instead have something like:

_______________________________________________________________________________________ test_equal[http://en.google.com/*-http://en.google.comm/*-True] _______________________________________________________________________________________

text1 = 'http://en.google.com/*', text2 = 'http://en.google.comm/*', equal = True

    @pytest.mark.parametrize('text1, text2, equal', [...])
    def test_equal(text1, text2, equal):
        pat1 = urlmatch.UrlPattern(text1)
        pat2 = urlmatch.UrlPattern(text2)

>       assert (pat1 == pat2) == equal
E       AssertionError: assert (equals failed
E         qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.com/*')   qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.comm/*') ) == True

tests/unit/utils/test_urlmatch.py:690: AssertionError

or if that's too hard to do (didn't look into the code so far), perhaps don't show decorators at all:

text1 = 'http://en.google.com/*', text2 = 'http://en.google.comm/*', equal = True

    def test_equal(text1, text2, equal):
        pat1 = urlmatch.UrlPattern(text1)
        pat2 = urlmatch.UrlPattern(text2)

>       assert (pat1 == pat2) == equal
E       AssertionError: assert (equals failed
E         qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.com/*')   qutebrowser.utils.urlmatch.UrlPattern(pattern='http://en.google.comm/*') ) == True

tests/unit/utils/test_urlmatch.py:690: AssertionError

which feels like much less noise.

This came up in the recent pytest chatter talking to @asottile and @RonnyPfannschmidt (and others, but I don't know your GitHub nicks, sorry!).

IIRC @asottile also mentioned a similar annoyance with long docstrings, which I don't have an example for off-hand. Not sure if it makes sense to complete redact docstrings (after all, they could be useful!), but maybe we can do so after a couple of lines?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the parametrized test example in tests/unit/utils/test_urlmatch.py and trace how pytest formats traceback source context for @pytest.mark.parametrize cases. Compare the requested reduced output with the current traceback, including the optional docstring behavior; done means failing output omits irrelevant parameter data while retaining the failing values and useful test context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.