pytest-dev / pytest-dev/pytest

Presence of a plugin causes doctests a shared namespace to fail traceback with assertion

Open
#8,059 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: doctests status: help wanted topic: rewrite
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

In jaraco/jaraco.test#3, I've described an issue that appears to implicate some of the most edgy features of pytest and Python, in particular:

  • assert rewrite
  • doctests
  • namespace packages
  • plugins

Oh, joy.

In summary, the issue seems to be that:

The presence of an empty module as a plugin in a namespace shared by the package under test causes assert exceptions in doctests to be re-written, causing the output from those doctests to fail where they wouldn't otherwise, even if the plugin is disabled with -p no:....

As you can see from the tox execution in the downstream report, I'm using pytest 6.1.2 on Python 3.9.0 with jaraco.test 4.0.1 (the implicated plugin) on macOS 11.0.1 (latest stable everything as of this writing).

I don't yet have a minimal example, though you can check out jaraco/jaraco.itertools@e9f23a4 and run tox -- -k assert to see the error.

I expect (though haven't verified) that any of the following remedies might be possible:

  • Move the plugin to a package not sharing the jaraco namespace. I was considering creating pytest-enabler anyway.
  • Replace the bare assert with a raise AssertionError. I'll test that soon.
  • Update the test expectation to expect the rewritten assert output.
  • Replace the doctest with a unit test.

None of these workarounds would address the root cause, but instead avoid the confounding factors leading to the failed expectation.

I recognize this issue is probably too obscure to expect it to be fixed upstream, but I could really use some advice on how to proceed with an investigation. In particular, can you explain or point me to the implementation where doctests get special treatment for assert-rewrite (or vice-versa), especially any code that might be relevant for selectively affecting the modules-under-test or plugins?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by checking out jaraco/jaraco.itertools at e9f23a4 and run tox -- -k assert to reproduce the doctest failure. Then trace pytest's doctest handling and assertion rewriting, focusing on the effect of an empty plugin module in the shared jaraco namespace and whether disabling the plugin changes it. Done means documenting a minimal reproduction and a clear root cause or upstream direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.