pytest-dev / pytest-dev/pytest

Populate output xml file after every test on a suite is completed

Open
#12,350 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: junitxml type: proposal
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

What's the problem this feature will solve?

I have been running tests suites on automated regressions via Jenkins using pytest for a while. Usually it runs without issues but from time to time due to timeouts an interrupt will kill the job and the output xml file will not be generated even thought some tests have been successfully run.

Is there a chance to extend/generate the output xml file after every test is completed? So if the suite face a timeout sigterm we still will be able to get the results for the tests that run without issues until that point.

I've run through the code (I'm not an expert on pytest) but I think it is related to this function.

Example: Lets say that we have a test suite with 10 tests, sometimes due timeout the job receive a sigterm that kills it at some point, for example, lets say on the 7th test , but all 6 previous tests have been run successfully. At this point the output xml file is not been created so the info for the previous 6 tests is lost.

Describe the solution you'd like

I would like to know if there is a flag o way to make the xml output generated every time a tests runs to avoid the issue of loosing information (specially if it is a big suite with far more tests and it takes a lot more of time to run).

I'm quite aware that this process (if possible or implemented) could add more time to the overall run time, but it will be appreciated and a reasonable trade off (at least in my case) instead of having to run a big suite of tests from scratch if a percentage or big part of it is already completed.

Additional context

Current versions related to pytest being used on my test environment:

pytest-randomly==3.4.0
pytest-repeat==0.8.0
pytest-timeout==1.3.4
pytest==6.2.5

I'll be glad to hear any suggestion from the pytest team or contribute with a PR for this one if needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked function in src/_pytest/junitxml.py around line 648 and trace when the output XML is created and finalized. Review the existing JUnit XML behavior and determine how interrupted runs could preserve completed-test results; done means a suite terminated by SIGTERM leaves a usable XML report for tests completed before termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.