microsoft / microsoft/playwright-pytest

retain-on-failure discards trace/video if teardown/setup fails in fixtures

Open
#117 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

p3-collecting-feedback
Dominant language
Python
Stars
563
Forks
90
Avg merge
11m
Merged PRs (30d)
1

Description

From what I can see, if a fixture fails in setup/teardown, the pytest --output .out --tracing retain-on-failure discards the trace. And at least sometimes the user will want the trace to be kept.

The code only looks at whether the rep_call has failed. https://github.com/microsoft/playwright-pytest/blob/main/pytest_playwright/pytest_playwright.py#L224-L226

But that ignores issues in the setup/teardown, e.g. pytests the example code

# content of conftest.py

import pytest


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    # set a report attribute for each phase of a call, which can
    # be "setup", "call", "teardown"

    setattr(item, "rep_" + rep.when, rep)


@pytest.fixture
def something(request):
    yield
    # request.node is an "item" because we use the default
    # "function" scope
    if request.node.rep_setup.failed:
        print("setting up a test failed!", request.node.nodeid)
    elif request.node.rep_setup.passed:
        if request.node.rep_call.failed:
            print("executing test failed", request.node.nodeid)

So if a user writes a fixture, that does something with a page in it's setup or teardown, and that setup or teardown code fails, the trace/video will be discarded. That's because at this point pytest will set rep_setup.failed (or rep_teardown) to be true, rather than rep_call.failed, I believe. Docs for the runtest_protocol

@pytest.fixture()
def logged_in(page: Page) -> Page:
    """Log in"""
    log_in_flow(page)
    yield page
    log_out_flow(page)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pytest_playwright/pytest_playwright.py at the trace-retention logic around lines 224-226, then review pytest's runtest_protocol and the fixture report example linked in the issue. Done means traces and videos are retained when fixture setup or teardown fails, not only when rep_call.failed is true.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.