pytest-dev / pytest-dev/pytest

`assert str1 == str2` takes forever with long strings that differ by a short prefix

Open
#8,998 34 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: rewrite type: performance
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Environment and versions:

platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/edited, configfile: pytest.ini
plugins: xdist-2.3.0, timeout-1.4.2, forked-1.3.0
timeout: 10.0s
timeout method: signal
timeout func_only: False

test-and-output.zip

To reproduce: download test.py and run pytest test.py --timeout 10 -vv

EXPECTED: both test methods (test1 and test2) fail quickly with assert False
OBSERVED: test1 fails quickly and test2 timeouts after 10 seconds (or runs forever if no --timeout flag passed)

The difference between test1 and test2 is simple

  • test1 does x = ' long string' == 'long string'; assert x and behaves as expected, but
  • test2 does straight assert ' long string' == 'long string' and pytest substitutes '==' with its logic that, apparently, has the bug.

The "long string" part of the literal strings is the same four times: twice in test1 and twice in test2.
The two strings that are compared with == both times differ by one leading space character only.
The actual strings in test.py are a bit longer (but not crazy long, some 159000 ANSI characters/bytes), but they differ just by one leading space still.

Full output attached as output.txt, that includes the interesting stack trace which hints at where the execution is happening at the moment when the execution is cancelled by the timeout. I will paste that stack trace here for convenience:

../../.local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:485: in _call_reprcompare
    custom = util._reprcompare(ops[i], each_obj[i], each_obj[i + 1])
../../.local/lib/python3.9/site-packages/_pytest/assertion/__init__.py:141: in callbinrepr
    hook_result = ihook.pytest_assertrepr_compare(
../../.local/lib/python3.9/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../../.local/lib/python3.9/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../.local/lib/python3.9/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../../.local/lib/python3.9/site-packages/_pytest/assertion/__init__.py:179: in pytest_assertrepr_compare
    return util.assertrepr_compare(config=config, op=op, left=left, right=right)
../../.local/lib/python3.9/site-packages/_pytest/assertion/util.py:153: in assertrepr_compare
    explanation = _compare_eq_any(left, right, verbose)
../../.local/lib/python3.9/site-packages/_pytest/assertion/util.py:176: in _compare_eq_any
    explanation = _diff_text(left, right, verbose)
../../.local/lib/python3.9/site-packages/_pytest/assertion/util.py:241: in _diff_text
    explanation += [
../../.local/lib/python3.9/site-packages/_pytest/assertion/util.py:241: in <listcomp>
    explanation += [
/usr/lib/python3.9/difflib.py:912: in compare
    yield from g
/usr/lib/python3.9/difflib.py:979: in _fancy_replace
    cruncher.ratio() > best_ratio:
/usr/lib/python3.9/difflib.py:651: in ratio
    matches = sum(triple[-1] for triple in self.get_matching_blocks())
/usr/lib/python3.9/difflib.py:486: in get_matching_blocks
    i, j, k = x = self.find_longest_match(alo, ahi, blo, bhi)

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the timeout with the attached test.py using pytest test.py --timeout 10 -vv. Start in _pytest/assertion/util.py at _diff_text and trace the call into difflib; done means both long-string comparisons fail promptly rather than timing out, with the existing assertion behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.