pytest-dev / pytest-dev/pytest

Tweak `--tb=auto` to be equivalent to `--tb=short`

Open
#9,731 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: proposal
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

There's earlier discussion (and the origin of --tb=auto) over in #364.

I'm not sure that the behaviour of --tb=auto is the best default behaviour that we can provide our users:

  1. It implies un-proportional importance of the first and last frame in a traceback, since they're the only ones whose functions are printed
    • A significant amount of time, the last frame is very unhelpful in a traceback because it's down in 3rd-party or standard-library code.
    • As projects get larger, the majority of code (and the majority of test risk) is in the code being orchestrated, not the test itself. So, the first frame (the test itself) becomes less likely to be the cause of the failure than the rest of the code running in the test.
  2. It can be tricky and a little time-consuming to find the "meat of the traceback", especially if the first/last frames in the traceback are within very large functions (I promise I didn't write them! I'm just the unlucky new maintainer 😛)

If we adopt --tb=short to be the default instead:

  1. It's more familiar, because it's similar to a regular Python exception traceback (though more concise, which I really appreciate. Great work team!)
  2. The entire stack is visible at once without any scrolling, which makes it easier to understand the shape of the failure.
Alternative Solutions

One way to improve the current situation without anything too radical would be to expand the last first-party code frame in the stack, leaving all 3rd-party and standard-library frames collapsed by default. However, it can be tricky to identify "first-party vs third-party", especially when sys.path modifications are happening (hello from Firefox's build tools! 👋 😄 ).

Additional context

I have a bug reported downstream with additional rationale, some of it project-specific.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the earlier discussion in #364 and compare the current --tb=auto and --tb=short behavior in pytest. This issue needs an agreed traceback policy before implementation; done would include the chosen behavior and its corresponding project changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.