pytest-dev / pytest-dev/pytest

Unequal location hints point at equal sections

Open
#4,765 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

I've encountered a situation where pytest incorrectly reports the character positions of mismatched characters in two strings being compared. Consider this test:

import textwrap

expected = textwrap.dedent("""
    --w-r---wx 1 0     0     999999 2019-02-01 bar
    drwxr-xr-x 2 0     0         64 2019-02-01 biz
    -rw-r--r-- 1 0     0        100 2019-02-01 foo
    """).lstrip().format(**locals())

actual = textwrap.dedent("""
    --w-r---wx 1 0     0     999999 2019-02-01 bar
    drwxrwxr-x 2 0     0         64 2019-02-01 biz
    -rw-rw-r-- 1 0     0        100 2019-02-01 foo
    """).lstrip().format(**locals())


def test_equal():
    assert expected == actual

Running pytest on that file emits:

draft $ pip-run -q pytest -- -m pytest -vv test-cmp.py                                                                                                                                   
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.7.2, pytest-4.2.0, py-1.7.0, pluggy-0.8.1 -- /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
cachedir: .pytest_cache
rootdir: /Users/jaraco/draft, inifile:
plugins: xonsh-0.8.9
collected 1 item                                                                                                                                                                        

test-cmp.py::test_equal FAILED                                                                                                                                                    [100%]

======================================================================================= FAILURES ========================================================================================
______________________________________________________________________________________ test_equal _______________________________________________________________________________________

    def test_equal():
>       assert expected == actual
E       AssertionError: assert '--w-r---wx 1...9-02-01 foo\n' == '--w-r---wx 1 ...9-02-01 foo\n'
E           --w-r---wx 1 0     0     999999 2019-02-01 bar
E         - drwxr-xr-x 2 0     0         64 2019-02-01 biz
E         ?        ---
E         + drwxrwxr-x 2 0     0         64 2019-02-01 biz
E         ?  +++
E         - -rw-r--r-- 1 0     0        100 2019-02-01 foo
E         ?        ---
E         + -rw-rw-r-- 1 0     0        100 2019-02-01 foo
E         ? +++

test-cmp.py:17: AssertionError
=============================================================================== 1 failed in 0.06 seconds ================================================================================

The failure is expected, but the reporting that it emits is not. As you can see, the ^^^ hints are pointing to sections that are the same in the actual and expected results.

As I'd never encountered this before, I originally thought it was a unittest bug, but this example doesn't use unittest at all.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproducer in test-cmp.py with pytest and compare the displayed hints with the differing characters in the two multiline strings. Trace pytest's assertion-reporting path for string comparisons. Done means the location hints identify the mismatched sections rather than equal sections, with the reproducer still passing as an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.