pytest-dev / pytest-dev/pytest

[junitxml] logic for handling interlaced test reports in the pytest_runtest_logreport hook is xdist-specific but it could be xdist-agnostic (generic)

Open
#14,078 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: junitxml type: enhancement
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Description

The src/_pytest/junitxml.py::pytest_runtest_logreport() hook implemented in the pytest built-in junitxml plugin has logic to handle interlaced test reports but that logic is too xdist-specific because it expects worker_id and item_index attributes in test reports that are set only by the pytest-xdist plugin. This part of code could be fully xdist-agnostic and depending only on uniqueness of report.nodeid attribute.

This is causing issues when generating JUnit XML report file with the pytest-cocotb plugin that allows to use pytest as regression manager for cocotb tests.

Captured stdout and stderr in report.capstdout and report.capstderr where report.when in ("setup", "call") are not included in JUnit XML report file when junit_logging is set to all|system-err|system-out.

I can prepare a Pull Request and refactor that hook. And add some unit tests.

Related issue: https://github.com/cocotb/cocotb/issues/5226

pip list

Package        Version
-------------- -------------------
cocotb         2.1.0.dev0+e4b4975d
find-libpython 0.5.0
iniconfig      2.3.0
packaging      25.0
pluggy         1.6.0
pygments       2.19.2
pytest         9.0.2

OS

Example

Git clone repository https://gitlab.com/tymonx/cocotb-junit-testing-ci-reports.git:

git clone https://gitlab.com/tymonx/cocotb-junit-testing-ci-reports.git

Change current working directory to it:

cd cocotb-junit-testing-ci-reports

Create environment using container image with GHDL simulator:

podman run -it --rm --workdir "$(pwd)" --volume "$(pwd):$(pwd):rw" --userns keep-id registry.gitlab.com/tymonx/ghdl:latest bash

Create Python virtual environment:

uv venv

Enter created Python virtual environment:

. .venv/bin/activate

Install Cocotb + Pytest:

uv pip install git+https://github.com/tymonx/cocotb.git@chore/xunit-reporter pytest

Run tests and generate JUnit XML report file junit.xml:

SIM=ghdl HDL_TOPLEVEL_LANG=vhdl WAVES=1 pytest -p cocotb_tools.pytest.plugin --override-ini=junit_logging=all --override-ini=junit_family=xunit1 --junit-xml=junit.xml

Generated JUnit XML report file junit.xml should contain COCOTB_RANDOM_SEED string in captured stderr:

grep COCOTB_RANDOM_SEED junit.xml
  • a detailed description of the bug or problem you are having
  • output of pip list from the virtual environment you are using
  • pytest and operating system versions
  • minimal example if possible

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/_pytest/junitxml.py at pytest_runtest_logreport() and read the report fields defined in src/_pytest/reports.py, especially nodeid, capstdout, and capstderr. Reproduce the issue with the provided cocotb command, then add unit tests covering interlaced reports and verify that the captured output appears in junit.xml for the requested junit_logging modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.