pytest-dev / pytest-dev/pytest

Fix double-counting of subtest failures in test summary

Open
#13,986 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

What's the problem this feature will solve?

When running tests using subtests in pytest, failures are being double-counted in the final summary. For example, if 1 test is being run with 3 subtests where 2 of the subtests pass and 1 fails, the summary incorrectly shows "2 failed, 2 subtests passed" instead of properly separating the counts.

This makes it difficult to understand the actual test results at a glance, as subtest failures are counted both as regular test failures and as subtest failures, leading to inflated failure counts.

Describe the solution you'd like

The summary should properly separate subtest failures from regular test failures. For the example above (1 test with 3 subtests, 1 subtest fails, 2 pass), the summary should show:
============= 1 failed, 1 subtests failed, 2 subtests passed in 0.02s =============

This provides clear visibility into:

  • How many top-level tests failed (1 failed)
  • How many subtests failed (1 subtests failed)
  • How many subtests passed (2 subtests passed)

This makes it much easier to understand the test results and identify whether failures are in subtests or in the main test logic.

Alternative Solutions

Currently, users need to manually parse the detailed output to understand the actual failure counts, which is error-prone and time-consuming. There's no workaround that provides the same clarity as fixing the counting logic.

Additional context

This is a feature that improves the accuracy and clarity of pytest's test summary output when using subtests. The change ensures that subtest failures are counted in their own category ("subtests failed") rather than being double-counted as both regular failures and subtest failures.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating pytest's test-summary counting logic and the code paths that handle subtest outcomes. Reproduce the example with one test and three subtests, then trace how the failed subtest enters both counts. Done means the summary separates top-level failures, subtest failures, and subtest passes without double-counting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.