pytest-dev / pytest-dev/pytest

Assertion rewriting in doctests

Open
#5,939 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: doctests topic: rewrite type: proposal
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

As a feature request, it would be convenient if doctests did the same assertion rewriting as regular tests. For example, here is the same failing test with and without doctests:

a = False
b = True

def bar():
    """
    >>> assert a == b
    """
    pass

def test_bar():
    assert a == b

Result of running with pytest version 5.2.1:

$ pytest --doctest-modules
====================================== test session starts =======================================
platform darwin -- Python 3.6.3, pytest-5.2.1, py-1.5.2, pluggy-0.13.0
rootdir: .../pytest_test
collected 2 items

test_foo.py FF                                                                             [100%]

============================================ FAILURES ============================================
_____________________________________ [doctest] test_foo.bar _____________________________________
005
006     >>> assert a == b
UNEXPECTED EXCEPTION: AssertionError()
Traceback (most recent call last):

  File ".../python3.6/doctest.py", line 1330, in __run
    compileflags, 1), test.globs)

  File "<doctest test_foo.bar[0]>", line 1, in <module>

AssertionError

.../pytest_test/test_foo.py:6: UnexpectedException
____________________________________________ test_bar ____________________________________________

    def test_bar():
>       assert a == b
E       assert False == True

test_foo.py:11: AssertionError
======================================= 2 failed in 0.07s ========================================

The regular test has assert rewriting that makes the problem easy to diagnose, while the doctest requires the user to go back and add debugging statements (which themselves are a little tricky to get right in doctests).

Apologies if this has been discussed before -- I guessed it would have been, but couldn't find a an issue about it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading pytest's doctest integration and running the reported pytest --doctest-modules example alongside the regular test. Trace how doctest assertions are collected and reported; done means failing doctest assertions provide useful rewritten diagnostics comparable to regular pytest assertions, with relevant coverage added.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.