pytest-dev / pytest-dev/pytest

running counter of test results at -2

Open
#12,800 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: proposal
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Currently, at both -1 and -2 the test cases reporting uses the "sea of dots".

For test suites which are both very large and relatively slow, this representation is not really helpful as counting dozens of dots is difficult, and the only easy to understand progress reporting is at end of line.

The "default" progress reporting yields a more precise view as it checkpoints the percentage every file, but that's at the cost of enormous amounts of space if the suite uses a large number of files (which is a good reason to use the dots).

I would propose an alternative representation, which is a lot terser and easier to track for slow test suites (though not necessarily for very small or fast ones): display a running counter in the same format as the summary stats line (maybe with the number of selected tests added) as the sole tally line e.g.

collected 5 items
1 passed / 5 selected

which on the next test would become

2 passed / 5 selected

and on failure

1 failed / 2 passed / 5 selected

This provides for a much improved "at a glance" summary when running tests interactively.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating pytest's -1 and -2 progress-reporting implementation and the code that formats summary statistics. Compare the existing dot and end-of-line modes while running a large, slow test suite. Done means an interactive running tally reports selected, passed, and failed tests without the sea of dots or per-file percentage output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.