robotframework / robotframework/robotframework

Unable to filter tests (--test) when re-running failed tests (--rerunfailed)

Open
#2,798 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.9k
Forks
2.6k
Avg merge
1d 18h
Merged PRs (30d)
10

Description

Filtering tests when rerunning failed tests does not work.

The attached file contains two tests that will always fail.

$ pybot --version
Robot Framework 3.0.2 (Python 2.7.14 on linux2)

$ pybot --output output_full.xml --log NONE --report NONE one.robot.txt
==============================================================================
One.Robot
==============================================================================
HDFS Test                                                             | PASS |
------------------------------------------------------------------------------
Hive Test                                                             | PASS |
------------------------------------------------------------------------------
Abracadabra Test                                                      | FAIL |
AssertionError
------------------------------------------------------------------------------
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
4 critical tests, 2 passed, 2 failed
4 tests total, 2 passed, 2 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output_full.xml

$ pybot --rerunfailed output_full.xml --test '*alakazam*' --output output.xml --log NONE --report NONE one.robot.txt 
==============================================================================
One.Robot                                                                     
==============================================================================
Abracadabra Test                                                      | FAIL |
AssertionError
------------------------------------------------------------------------------
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
2 critical tests, 0 passed, 2 failed
2 tests total, 0 passed, 2 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output.xml

A regular run while only filtering test cases works fine (without --rerunfailed):

$ pybot  --test '*alakazam*' --output output.xml --log NONE --report NONE one.robot.txt 
==============================================================================
One.Robot                                                                     
==============================================================================
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output.xml

one.robot.txt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the attached one.robot.txt file using the shown --rerunfailed and --test commands, then compare it with the regular filtered run. Trace how failed-test reruns and test-name filtering are combined; done means rerunning output_full.xml with --test 'alakazam' executes only Alakazam Test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.